What it is
The European Central Bank defines data lineage as information about the movement and transformation of data from front, at capture, to end. It covers extraction, transformation and loading.
Impact analysis uses the same record forwards: starting from a source, a field or a job, it shows every report and dataset that depends on it.
OpenLineage is an open standard for collecting lineage from jobs as they run. It records how datasets come into being, not just that relationships exist between them.
Why it matters
The European Central Bank expects banks to hold complete and up-to-date lineage at the level of the data attribute. In its thematic review, none of the significant institutions sampled had fully followed the Basel principles for risk data. That finding concerns the principles as a whole, not lineage alone.
DII has read evidence for banking only. Outside it, the case rests on the practical value of knowing what a change will break.
The problem it addresses
DII editorial view
When a reported figure is questioned, or a source is about to change, the organisation needs to know what feeds what. Without lineage that knowledge sits in people's heads or in diagrams that are out of date, and is rebuilt by hand each time it is needed.
Business and organisational relevance
DII editorial view
Lineage is asked for by auditors and supervisors and used by engineers. The organisational question is who keeps it current, because lineage that is out of date is trusted wrongly.
Practical use cases
- Showing a supervisor or auditor where a reported figure came from
- Finding what a schema change or a decommissioning will break
- Tracing a data incident back to its cause
Benefits and opportunities
- Regulatory evidence
- Safer change
- Faster incident triage
Risks, limitations and dependencies
- Lineage drawn by hand goes out of date
- Attribute-level lineage across a whole estate is costly; scope matters
- Lineage tools see only the systems they are connected to
Prerequisites for adoption
DII editorial view
- A defined scope; attribute-level lineage across a whole estate is costly
- Systems the lineage tooling can actually connect to
- Someone responsible for keeping it current
Implementation considerations
- Start with the critical data elements. Trace the fields behind key reports first, at attribute level, before attempting the whole estate.
- Collect lineage from running jobs where you can. Lineage drawn by hand goes out of date; lineage collected as jobs run stays current.
- Keep it current. An out-of-date lineage is worse than none, because it is trusted.
The DII Radar assessment
Catalogue status
Not yet assessed. DII has not published a Radar assessment for Data Lineage and Impact Analysis. It is listed in the catalogue so that it can be explained. No relevance, direction, strategic impact or enterprise adoption is given for it, and it is not plotted on the Radar.
What to consider next
DII editorial view
- Start with the critical data elements behind key reports
- Collect lineage from running jobs where you can
- Keep it current; an out-of-date lineage is trusted wrongly
Evidence and references
The sources DII read for this explainer. Each entry gives what the record holds; nothing is added.
- OpenLineage documentation: introduction and Object Model (version 1.53.0) View source for reference 1 (external site)
- Guide on effective risk data aggregation and risk reporting Introduction, section 3.4 and footnote 23 were read. View source for reference 2 (external site)