Where to Put Data Quality Checks in a Pipeline, and Which Ones Earn Their Keep
Where a check stands in a pipeline decides what it can catch and what should happen when it fails. Most checks should warn and a few should stop the pipeline. A check that has never failed, or that fails and is ignored, should be removed.
Data quality checks are easy to add and awkward to remove, so pipelines accumulate them. A team that has been burned adds a test for the fault that burned it. A new tool makes tests cheap, and every column gets three. In time the pipeline runs a great many checks, some fail every night, nobody looks, and the one failure that matters is lost among the ones that do not.
The opposite condition also exists: no checks, and every fault found by a user. This article takes a position between them. It describes where in a pipeline checks belong, which should block and which should warn, how to judge whether a check is worth keeping, and what to do when one fails. The argument is DII’s judgement, built on published guidance, and it is opinionated on purpose.
What a check is, and what it is for
A check is an assertion about data that can be evaluated automatically. dbt’s documentation, describing its own implementation, puts it simply: “Data tests are assertions you make about your models and other resources in your dbt project”1. It also notes the form of the result: “Data tests return a set of failing records”1. Great Expectations, another open-source project, describes itself in similar terms, as “a framework for describing data using expressive tests and then validating that the data meets test criteria”2.
The purpose of a check is to stop bad data reaching a user, or to tell someone quickly that it has. Quality here means what the UK Government Data Quality Framework says it means, fitness for purpose3. A check should therefore trace back to a purpose: some use of the data that would be harmed if the assertion were false. A check with no such use behind it measures something nobody needs.
Where along the pipeline a check can stand
The Government Data Quality Framework states the principle that organises everything else: “Quality assessment and assurance should take place at each stage of the lifecycle. The measures used will change at each stage”3. The second sentence is the important one. The stages of a pipeline call for different checks, and repeating the same checks at every stage adds cost without adding protection.
Positions for checks in a pipeline
| Position | What to check | Typical checks | Usual response to failure |
|---|---|---|---|
| On arrival | Did the source deliver what was agreed? | Arrived on time; expected volume; expected schema; required fields present | Stop: do not process a bad delivery |
| After transformation | Did our logic do what we intended? | Keys unique; joins did not multiply or drop rows; totals reconcile to input | Stop for structural faults; warn for the rest |
| Before publication | Is this fit for the people about to use it? | Business rules; reconciliation to a system of record; plausibility against history | Stop: hold publication and keep the last good version |
| In use | Is published data still behaving normally? | Freshness, volume and distribution over time | Warn: alert an owner |
On arrival
Checks at the entrance test the source, not the pipeline. They ask whether what arrived matches what was expected. This is the cheapest place to catch a fault, because nothing has yet been built on it, and the right response to a failed delivery is to stop and not process it.
Where the source is another team’s system, arrival checks are the consumer’s side of an agreement about what will be delivered. Writing that agreement down is the subject of how to implement data contracts between producers and consumers.
After transformation
Checks here test the team’s own logic. The classic faults are structural: a join that duplicates rows, a filter that drops more than intended, a key that is no longer unique. They are bugs, and the checks that catch them are closer to software tests than to data quality rules. Reconciling a total before and after a transformation is among the most valuable checks there is, because it catches a wide class of errors without needing to know which one.
Before publication
The last position before users is where business meaning is tested. Does the revenue total agree with the finance system? Is today’s figure within a plausible range of recent days? These checks need knowledge of what the data is for, and they are the ones worth agreeing with the people who use it. The acceptance criteria written for a report, as described in dashboard requirements: what to settle before anyone builds, are a ready source of them.
Publication is also where a failed check does most good. Holding back a bad load and leaving yesterday’s correct data in place turns an incident into a delay. That depends on the pipeline separating building from publishing, one of the properties described in what makes a data pipeline reliable.
In use
After publication, checks become monitoring: watching freshness, volume and the shape of values over time, to catch what the earlier positions missed. This is the territory of data observability: what to monitor and where to start.
Why checking each stage is not enough
One caution applies across all of these positions. Google’s Site Reliability Workbook warns that “if you measure data correctness only per stage, you could miss end-to-end data corruption bugs”4. Each stage can pass its own checks while the output is wrong, because each stage tests against its own assumptions.
The remedy is at least one check that spans the pipeline: a reconciliation of the final output against the original source or an independent system of record. If only one check could be kept, this would be the candidate.
The warning comes from a book about running software services at one company, in a chapter that does concern data pipelines, which makes it more directly applicable than most software guidance. Its limits should still be stated. It describes pipelines at a scale and with an engineering staff few organisations have, and it treats correctness as a service objective to be measured. It has nothing to say about which business rules to check or how to agree them with the people who use the data. That part of this article, and the judgement about which checks are worth keeping, is DII’s.
Block or warn
Every check needs a decision about what happens when it fails. There are two choices. A blocking check stops the pipeline. A warning lets it continue and tells someone.
Tools make the distinction explicit. dbt’s configuration, for example, gives each test a severity of error or warn, with error as the default5. The setting is easy. The judgement is which to choose.
Block when publishing bad data is worse than publishing nothing. That is true of structural faults that make everything downstream wrong, of data for financial or external reporting, and of anything that cannot easily be corrected once it has been used.
Warn when late data is worse than imperfect data. An operational dashboard with a few records missing an optional field is still useful. Stopping it harms users more than the fault does.
The Government Data Quality Framework recognises the underlying tension, noting that the needs of users “may result in trade-offs between different dimensions of data quality”3. Timeliness and completeness pull against each other, and the block-or-warn decision is where a team chooses between them for a given data set.
Two practical rules follow from how blocking checks behave. They are DII’s.
- Be sparing with blocks. Every blocking check is a way for the pipeline to stop at night. A team that is woken by blocks that did not matter will downgrade them all, including the ones that did.
- Never leave a warning without a reader. A warning that goes to a log nobody opens is decoration. Each needs a named owner who sees it and a stated expectation of what they do.
Which checks earn their keep
A check costs something: compute on every run, maintenance when the data legitimately changes, and attention each time it fires. A few questions test whether it repays that.
- What use of the data does it protect? If nobody can name one, remove it.
- Has it ever failed for a real reason? A check that has never failed in a year may be guarding against something that cannot happen. Uniqueness on a key the database already enforces is an example.
- When it fails, does anyone act? If the routine response is to rerun or ignore, the check is noise. Fix its threshold or delete it.
- Is the same fault caught elsewhere? Duplicate checks at several stages add cost and no protection.
Applying these to an existing pipeline is a useful exercise, and removals are the result to expect. A smaller set of checks that people respect protects data better than a large set they have learned to ignore.
The choice of what to check thoroughly should follow the importance of the data. Concentrating the strongest checks on the data elements the organisation has identified as critical, as set out in starting with critical data elements, is a better use of effort than uniform coverage of every column.
When a check fails
A check is only as good as what happens next. For each blocking check and each warning, three things should be written down beside it: who is told, what they look at first, and who decides whether to release the data anyway.
The third is the easiest to leave out. A blocking check will sometimes stop a pipeline for a reason that, on inspection, is acceptable: a genuine and explained change in the business. Someone needs the authority to override, and the override should be recorded with its reason. Without that route, people disable the check.
A starting set
For a pipeline with no checks, DII would begin with a small set and add others only as incidents or stated uses justify them.
- Arrival: the delivery is present, on time and of plausible size. Blocking.
- Arrival: the schema matches what is expected. Blocking.
- Transformation: primary keys are unique and not null. Blocking.
- Publication: a headline total reconciles to the source or a system of record within a stated tolerance. Blocking.
- In use: freshness of the published table. Warning, to a named owner.
The Data Pipeline Reliability Checklist covers the design properties these checks rely on, and the Data Engineering Maturity Assessment places testing practice alongside the other parts of a team’s work. No public DII Resource provides a register of checks.
Sources
- dbt Labs. Add data tests to your DAG (dbt Developer Hub).
- Great Expectations. GX Core overview (Great Expectations documentation).
- Government Data Quality Hub (UK Government). The Government Data Quality Framework. 3 December 2020.
- Sodt, R. and Maravić, I. (Google; The Site Reliability Workbook). Data Processing Pipelines (chapter 13 of The Site Reliability Workbook). 2018.
- dbt Labs. severity, error_if, and warn_if (dbt Developer Hub reference).
This article is general professional information. It is not legal or regulatory advice.