Close

HOW TO USE GENERIC TEMPLATES FOR DATA QUALIFICATION

Share

People are looking for shortcuts. Even if you are not a millennial (‘Y generation’) follows by rules like “faster is better”, as long as the quality is maintained – you will always end up choosing the shorter way to paradise.
If you need to do things regularly then you will search for an automatic solution. If you need to do things more of the same with a small difference then you will buy a copy machine and update only the delta on every copy. This saves not just time but ensure quality since there is almost no room for mistakes.
Simple as it sounds, if it is not the same lady with a different shirt, you need to return to your starting point and start from the beginning.

Testing data is a good example of testing almost the same things over and over again. With quilliup you can build a set of tests to compare tables, files or validate data. You can use variables in a predefined rule or within a SQL script and make the test generic. You can also use a management table that will monitor what was already tested and what is in the queue. At the end of the day you will have less tests to maintain and hopefully healthier routines.

How quilliup does it?

Imagine you have a large number of tables (or files, or any of your data platform entities) with similar characteristics of the data structure. You would like to go over all of these tables and to execute integrity checks for each of the tables in order to make sure the data is accurate and complete. Instead of going through each and every field and test them one by one, quilliup enables you to define your tests using variables and create generic tests. Then, by creating a set of rules and flow of tests, the system can run iteratively for each of your variables.

An example of a generic flow which runs iteratively and updates the status according to the test result.

At the end of the process, you can get all the results report to your email and also log these results to your database to track the changes. With this generic implementation you are saving time and you can maintain your tool easier and now you just need one trigger to activate the entire process and that’s it.

In the table above you can find an example of the generic flow results. You can see the result (Passed\ Warning\ Failed) for each entity in each cycle.