Data Quality Testing in Pipelines

What it is

A test is a statement about the data that can be checked automatically: a column is never empty, a value falls in a range, a total reconciles. Tool documentation describes data tests as assertions about models and other resources.

Where the test runs matters as much as what it checks. UK government guidance defines quality as fitness for purpose, so a check belongs next to the purpose it protects.

Why it matters

Quality is fitness for purpose, so a check belongs next to the purpose it protects. Tests as code are now a standard feature of the common transformation and validation tools.

DII infers adoption from that: the practice is a standard feature of widely used tools. No source it has read measures how many organisations use it.

The problem it addresses

DII editorial view

Quality problems are usually found by the people reading the output, after decisions have been made on it. Checks run afterwards, or by hand, arrive too late. Tests written as code run every time the pipeline runs, at the point where the data is about to be used.

Business and organisational relevance

DII editorial view

The organisational question is what happens when a test fails. Someone has to own each test, and the business has to accept that a pipeline will sometimes stop instead of delivering wrong figures on time.

Practical use cases

• Stopping a load when a critical field is missing
• Warning when a volume falls outside its normal range
• Checking that figures reconcile between a source and a report

Benefits and opportunities

• Bad data is caught before it reaches a report or a model
• Expectations about data are written down and versioned with the code

Risks, limitations and dependencies

• Tests with no owner and no agreed response
• So many warnings that real failures are missed
• Testing what is easy to test and not what the data is used for

Prerequisites for adoption

DII editorial view

• Pipelines under version control, so tests live with the code
• An owner for each test and an agreed response when it fails
• Business acceptance that a pipeline will sometimes stop instead of delivering wrong figures on time

Implementation considerations

• Give every test a severity and an owner. A test nobody is responsible for will be ignored when it fails.
• Decide in advance what stops the pipeline. Agree which failures halt delivery before the first one happens.
• Put checks next to the purpose. Test the data where it is about to be used, for the use it is about to have.

The DII Radar assessment

Approved DII Radar assessment

DII assesses Data Quality Testing in Pipelines as High relevance, with direction Stable. Its strategic impact is assessed as High and its enterprise adoption as Scaling. It is one of the ten entries on the current Radar.

DII relevance is our assessment of current significance and momentum, based on evidence, market development and adoption. Direction is the observed trajectory of relevance and adoption. Strategic impact is the potential or current significance to enterprise data capability, and enterprise adoption is the extent of current real-world enterprise adoption; together they set the position on the Radar. The Radar is not a prediction, vendor comparison or performance ranking.

Evidence basis: UK Government Data Quality Framework (2020); dbt data tests documentation; Great Expectations documentation (tool documentation).

See Data Quality Testing in Pipelines on the Radar →

What to consider next

DII editorial view

• Give every test a severity and an owner
• Decide in advance which failures stop the pipeline
• Test what the data is used for, not only what is easy to test