During an ETL process with executions, which normally runs consistently as part of a daily process, tracking and monitoring can become a bottleneck that hinders the purpose of analyzing the data efficiently.
A monitor is a visualization tool that presents data to showcase performance statistics, benchmarks, and comparisons. For example, when an existing nightly execution which is meant to run for an hour as part of an ETL process mistakenly ran for 3 hours each night for a week, the monitor will point this out. It’s critical to see and to analyze these types of biases in the ETL process, and find the core issues in the system to solve them.
Check out your running duration with comprehension to other executions in order to track running times.
How to build a monitor with high effectiveness for the ultimate situation report:
First of all, decide which are the most important metrics for your business to review. Since the monitor is an operative tool, it is mandatory to clear unnecessary interruptions.
To do that, you must define and understand the operational goal – this could be about the duration of executions, successes and failure rates, content of the rules, results etc.
The next step is to build the main scheme. The monitor is based on a scheme that contains several merged tables that are located in the product database. Merging tables into one scheme helps to present a wide, timely overview of executions across the system.
Thirdly, and arguably the most important step, will be the monitor’s visualization. The monitor has multiple visualization alternatives including templates, graphs and indications. Some of the most common options are:
- KPIs: Key Performance Indicators are commonly used as aggregational functions to present data such as how many tests failed on a certain day in comparison to the previous day.
- Bar Charts: this can be very helpful when measuring executions over time or benchmark performance of different executions to compare and review the duration or content(i.e. the amount of failed rows) of multiple tests.
- Pie Chart: these visuals are very helpful when you need insights into data distribution. For example, to review the percentage of successful executions.
The monitor tool was designed to provide access to data, with a very flexible interface and high functionality. quilliup recommends to customers that have access to Tableau to make the most of their pre-built monitor that connects to the data platform. This monitor contains real-time duration deviations and useful indicators about the ratio of failed executions alerts which updating the data in real-time regards the executions in quilliup. What’s more, all these indicators can be customized with your pre-defined percentage and deviation values.
So get started now! Create your customized data quality monitoring tools to make the most of your Quality Gates experience.
Track the most updated details regards all the executions in your environment.