How to Map Data Flows Across Systems and Keep the Map Current
A data flow map records where data comes from, what is done to it, where it rests and who uses it. Drawing one is the smaller task. The larger one is deciding who keeps it true after the project that needed it has ended.
A data flow map answers three questions about a dataset: where it originates, where it goes, and what depends on it. The need for one can arrive suddenly, with an audit, a migration or an incident, and at that point the people who could have answered from memory are busy with the audit, the migration or the incident.
This guide describes how to build a map that is useful at the moment of need and still accurate a year later. It covers how to scope the work, what to record for each flow and at what level of detail, how a hand-built map differs from automated lineage and where each belongs, and how to assign upkeep. The method draws on a data protection regulator’s guidance, a US government privacy framework and an open lineage standard, because each has had to say precisely what a map should contain.
A flow map and automated lineage are different things
The two can be discussed as if they were one, and they do different jobs.
A flow map is a document people maintain. It shows business-level movement: this system sends customer orders to that one, nightly, for this purpose. It can include movements that no pipeline tool will ever see, such as a spreadsheet emailed to a supplier.
Automated lineage is metadata that pipelines emit as they run. The OpenLineage project, an open standard hosted by the Linux Foundation’s AI and data foundation, describes itself as “an Open Standard for lineage metadata collection designed to record metadata for a job in execution”1. It records what actually ran, at the level of jobs and datasets.
A hand-built flow map and automated lineage compared
| Flow map | Automated lineage | |
|---|---|---|
| Produced by | People, from interviews and documents | Pipelines and query engines, as they run |
| Level of detail | Systems, datasets and business purpose | Jobs, tables and, where supported, columns |
| Covers manual and external transfers | Yes, if someone records them | No |
| Records purpose, owner and legal basis | Yes | No, unless added as extra metadata |
| Stays current without effort | No | Yes, for the pipelines that are instrumented |
| Best used for | Audits, impact assessment, onboarding, design reviews | Incident diagnosis, change impact, verifying the map |
The comparison is DII’s. Its practical consequence is that neither replaces the other. A map without lineage drifts from reality. Lineage without a map is accurate about pipelines and silent about why the data moves and who is entitled to it. The sound arrangement is a map people own, checked against lineage where lineage exists.
Scope by business process, not by system
The first decision is what to map, and the tempting mistake is to begin with a system inventory and try to document every interface. That produces a large diagram of uniform detail in which the three flows that matter are no easier to find than the three hundred that do not.
The UK Information Commissioner’s Office, in its guidance on documenting the processing of personal data, recommends the opposite order: “Start with the broadest piece of information about a particular processing activity, then gradually narrow the scope”2. For organisations it suggests the business function as the starting unit: “it makes sense for controllers to begin with a business function”2.
The advice transfers to any data, personal or not. Choose a process someone cares about, such as order to cash, a regulatory return or the monthly board pack, and follow its data from first capture to final use. The scope is then defined by a question a named person needs answered, and the map is finished when that question can be answered.
Three criteria help in choosing the first process: an event that makes it urgent, an executive who will use the result, and a size that can be mapped in weeks.
Capture five things for every flow
A flow is one movement of data from one place to another. For each, record five things.
- Source. The system or party the data comes from, and whether it is the system of record for that data or a copy.
- Transformation. What is done on the way: filtered, joined, aggregated, masked, re-keyed. A sentence is enough.
- Store. Where the data rests afterwards, and for how long.
- Consumer. Who or what uses it next: a report, a model, another system, an external recipient.
- Owner. The person who answers for this flow continuing to work and continuing to be justified.
A US government framework for privacy risk sets out a similar list. The NIST Privacy Framework asks organisations to inventory the systems that process data, their owners or operators, the data actions performed, the purposes of those actions and the data elements involved3. Its definition of the resulting artefact is a good test of completeness: “A data map shows the data processing environment and includes the components through which data are being processed or with which individuals are interacting, the owners or operators of the components, and discrete data actions and the specific data elements being processed”3.
Choose a grain you can maintain
Level of detail is where maps become unmaintainable. The NIST document is permissive on the point: “the level of detail may vary based on an organization’s needs”3. The ICO is stricter about what is too little. Documentation has to be done “in a granular and meaningful way”, and the guidance gives the failing case: “A generic list of pieces of information with no meaningful links between them will not meet the UK GDPR’s documentation requirements”2.
The useful reading of both is that the links matter more than the lists. A map that names every system and every dataset, and does not say which dataset goes where for what purpose, has recorded the nouns and left out the verbs.
DII’s working rule is to map at the level of dataset and system, not field and table, and to descend to fields only for data that carries a legal or financial consequence. Field-level definitions belong in a separate reference: the Data Dictionary holds them, and the map points to it. Which data carries that consequence is settled by classification, described in how to build a data classification scheme people will apply.
Gather it from people and from documents
Nobody holds the whole picture, so the map is assembled. The ICO’s guidance names three methods that work together: a questionnaire to the areas that process data, direct meetings with key functions, and a review of existing documents2. It is explicit about why breadth matters: “It is important that people across your organisation are engaged in the process; this can help ensure nothing is missed”2.
The third method deserves particular attention. The guidance advises teams to “Locate and review policies, procedures, contracts and agreements”, and observes that doing so helps to “compare and contrast intended and actual data processing activities”2. A contract that says data is sent monthly and an engineer who says it is sent nightly have between them identified something worth knowing.
In practice this means three passes. Interview the people who run the process and draw what they describe. Read the interface specifications, data-sharing agreements and pipeline code, and mark every disagreement. Then take the disagreements back to the people. The disagreements are what the exercise is for.
The Data Flow Mapping Template provides the structure for this record: one row per flow, with the five elements above, the classification of the data and a column for the date each row was last confirmed.
Add automated lineage where pipelines can report it
Where data moves through orchestrated pipelines, part of the map can be observed directly. OpenLineage “defines a generic model of dataset, job, and run entities”1, and its documentation explains the design choice behind that model: “OpenLineage cares how Datasets come into being, not just that relationships exist between them”1. A job is defined simply: “A Job is a process that consumes or produces Datasets”1. From enough observations, “a lineage graph can be created by weaving together observations of many Jobs across multiple platforms”1.
Three limits should be understood before relying on it.
- It sees only what is instrumented. The project lists integrations with specific schedulers and engines1. A flow through a tool with no integration, or through no tool at all, does not appear.
- It records what ran, which is evidence about the past. It does not say what is permitted or intended.
- It produces volume. A lineage graph of every job is as hard to read as a map of every interface, and needs the same scoping by process.
Used within those limits, lineage does two things a hand-built map cannot. It shows quickly what is downstream of a failing source, which is the first question in an incident and a recurring theme in what makes a data pipeline reliable. And it gives the map’s owner a way to check their document against what the pipelines actually did.
Where personal data is involved, some of this is required
For personal data, parts of a flow map are a legal record and not a matter of choice. Article 30 of the UK GDPR says that a controller “shall maintain a record of processing activities under its responsibility”, and lists what the record must contain, including “the purposes of the processing” and “the categories of recipients to whom the personal data have been or will be disclosed”4. The record must be “in writing, including in electronic form”4.
Two cautions. The legal record and a flow map overlap and are not the same document: the record is organised by purpose and says little about systems, and a flow map can be complete without stating a lawful basis. One exercise can feed both, which is the efficient way to do it. And the ICO states on the page cited here that its guidance is under review following changes made by the Data (Use and Access) Act, so the current position should be checked before relying on any detail2. This article is not legal advice.
Assign upkeep, or the map decays
A flow map is accurate on the day it is finished and less accurate every day after. The ICO makes the point about its own subject in terms that apply generally: “Keeping a record of your processing activities is not a one-off exercise”, and the record should be treated “as a living document that you update as and when necessary”2.
Good intentions do not achieve that. Three mechanisms do, and they are DII’s recommendation.
Tie updates to change, not to the calendar. A new interface, a retired system or a changed supplier is the moment the map goes out of date. Make the update a condition of the change being approved. Where significant choices are recorded as they are made, as described in architecture decision records for data teams, the map update belongs in the consequences of the record.
Give each flow an owner, not the map as a whole. A single custodian of a large map cannot know when a flow in another department has changed. The owner of a flow can.
Date every row. A column showing when each flow was last confirmed turns staleness from an invisible risk into a visible one. Rows unconfirmed for a year are the review list.
A periodic check still has a place, as a backstop. The natural occasion is an architecture review, where the map is both an input and something to be tested; running a data architecture review that changes something describes how.
A map kept this way is the working description of the estate: the place where the layers and decisions set out in modern data architecture: the layers, patterns and decisions that matter can be seen as they actually are.
Sources
- OpenLineage (LF AI & Data Foundation). OpenLineage documentation: introduction and Object Model (version 1.53.0).
- Information Commissioner’s Office (UK). How do we document our processing activities? (UK GDPR guidance: Documentation).
- National Institute of Standards and Technology (United States). NIST Privacy Framework: A Tool for Improving Privacy through Enterprise Risk Management, Version 1.0. 16 January 2020.
- UK Government (legislation.gov.uk). UK General Data Protection Regulation, Article 30: Records of processing activities. 2016.
This article is general professional information. It is not legal or regulatory advice.