Data Observability

What it is

Observability is the ability to understand a system from the outside, by asking questions about it without knowing its inner workings. Applied to data, it means watching the data itself and not only whether a job ran.

A common description, which comes from a vendor and not from a standard, names five things to watch: freshness, volume, schema, quality and lineage.

Why it matters

A data problem found by a consumer costs more trust than one found by the team responsible. Observability moves detection to the owning team.

One survey reports that 76% of organisations have formal programmes for data quality and pipeline observability. It was sponsored by two vendors and reported by one of them, so DII treats the figure with caution.

The problem it addresses

DII editorial view

A pipeline can run successfully and still deliver data that is late, incomplete or changed in shape. Job monitoring does not show this, so the first person to notice is often the reader of a report. Observability watches the data itself.

Business and organisational relevance

DII editorial view

Observability changes who finds out first. It gives the team that owns a pipeline the means to detect a problem, which only helps if that team is also the one expected to respond.

Practical use cases

• Detecting late or missing loads before a report is read
• Detecting an unannounced schema change
• Triage of a data incident, using lineage to find what is affected

Benefits and opportunities

• Earlier detection, by the owning team
• Evidence for incident review
• A shared measure of data reliability

Risks, limitations and dependencies

• Alert fatigue when too much is monitored
• Monitoring symptoms without ownership to act on them
• The five-part model is a vendor framing, not a standard

Prerequisites for adoption

DII editorial view

• A named owning team for each pipeline, so an alert has somewhere to go
• A view of which pipelines feed critical reports
• Agreement on what counts as worth alerting on

Implementation considerations

• Start with the pipelines behind critical reports. Monitor freshness, volume and schema there first, before extending coverage.
• Route alerts to the team that owns the pipeline. An alert nobody is responsible for will be ignored.
• Alert only on what someone can act on. Alerts that fire too often are second-guessed, skimmed or ignored.

The DII Radar assessment

Catalogue status

Not yet assessed. DII has not published a Radar assessment for Data Observability. It is listed in the catalogue so that it can be explained. No relevance, direction, strategic impact or enterprise adoption is given for it, and it is not plotted on the Radar.

What to consider next

DII editorial view

• Start with the pipelines behind critical reports
• Route alerts to the team that owns the pipeline
• Alert only on what someone can act on