Floemdata
Data & ETL

Data Quality: Checklist and Automation

A checklist for data quality: completeness, uniqueness, validity, consistency. We show which checks should be automatic and which need a person.

Floemdata · · 3 min read
A checklist with four rows: three ticked in green, one failed with a red cross

Data quality is not a 'check once and you are done' job; it consists of small checks that repeat after every load. The list below also shows which belongs to the machine and which to a person.

Automated checks (on every load)

  • Completeness: did the expected tables and periods arrive? Is the row count reasonable compared with the previous load?
  • Uniqueness: was the same document loaded twice? Are there duplicate records on the business key?
  • Validity: are dates real dates, are amounts numbers, are required fields filled, are codes in the defined list?
  • Consistency: does the header total match the sum of the lines? Does a customer code in one table exist in the other?
  • Freshness: when was the last successful load? Raise a warning if it is older than expected.
  • Reconciliation with the source: does the warehouse total match the source system's own report?

What needs human review

  • What a deviation caught by an automatic rule means for the business: a real change or an error?
  • The threshold of a new rule: a rule that is too tight produces false alarms, one that is too loose misses things.
  • Definition decisions: does a cancelled document count towards revenue, and which date is the reference?
  • The root of an error: is the data broken, or the process at the source?

Alarm fatigue

If every check raises an alarm, nobody looks. Put the checks in order of importance: blocking error, warning, information. Keep blocking errors for situations that would make the result wrong.

At Floemdata

In Floemdata Analytics, two principles overlap with this list (project design decisions): re-running a load does not insert the same record a second time, and a reconciliation deviation is not hidden but shown in the dashboard. The list above is a general recommendation; it does not mean that every item has an automatic counterpart in the product (not verified).

To get started, apply all six checks by hand to the data source of the report you use most, then pick the ones worth automating. Questions: info@floemdata.com.

← All guides