Handling Data Incidents: Severity, Triage and Learning From Failure
Software operations has a mature discipline for incidents: named roles, severity levels and written reviews without blame. Data teams can borrow most of it. Three things need adapting, because a data incident is not an outage: it can be silent, it spreads downstream, and the wrong numbers may already have been used.
Consider an illustration. A wrong figure reaches a board pack. Someone notices on the morning of the meeting. Several people begin investigating separately, two of them rerun the same pipeline at the same time, and the finance director hears about it from a colleague in the corridor. The data is corrected by noon. Nobody writes anything down, and the same fault later returns.
Almost all of that account is avoidable. Software engineering has spent two decades working out how to respond to failures in production, and the practices are published. This article adapts them for data teams: what counts as a data incident, how to grade severity, how to triage and communicate, and how to learn from what happened. It is an adaptation of published practice, and a later section says how far the adaptation can be trusted. It is not an account of DII’s own operations.
One boundary first. This article concerns operational incidents: data that is late, missing or wrong. A security incident, or any incident involving unauthorised access to or loss of personal data, falls under separate legal and organisational procedures with their own reporting obligations and time limits. If an incident might be one of those, the organisation’s security and data protection processes take over, and nothing here replaces them.
What makes a data incident different
The software practices assume a service that is up or down. Three features of data change the picture. They are DII’s analysis of where the borrowed model needs care.
Data incidents can be silent. A failed service returns errors. A pipeline that loads wrong data returns success. The incident may begin days before anyone knows, which is why detection is its own discipline, covered in data observability: what to monitor and where to start.
They spread. Wrong data in one table flows into everything built from it. Fixing the source is the beginning of the repair. Every downstream table, extract and report built during the incident has to be found and rebuilt.
The damage may already be done. A service outage ends when service is restored. A data incident may have produced a decision, a customer communication or an external submission based on wrong figures. Recovery therefore includes telling the people who used the data, which an outage does not require.
How far software incident practice applies
Every source used here was written about software services, and none of them discusses data pipelines. That needs saying plainly, because it limits what the sources can be taken to support.
Some of the practice transfers without strain, because it is about how people coordinate under pressure and has nothing to do with the kind of system that failed: naming one person to lead, keeping everyone else from changing things, declaring early, writing down what happened and reviewing it without blame.
Some of it does not transfer. Software incident practice is organised around restoring a service, and its measures of success are about availability. A data incident is not over when the pipeline runs again, because the wrong data it produced still exists downstream. Rolling back, the standard first move in software, does not un-publish a figure someone has already used. And the published criteria for declaring an incident assume that the failure is visible, which a data fault may not be.
The adaptation is therefore DII’s. The sources support the general discipline. They do not show that it works for data teams, and no study of data incident practice was found and read for this article. The parts below that concern containment, downstream repair and telling the people who used the data are DII’s additions, and should be read as reasoned proposals.
Severity: grade by consequence
A severity scale lets a team decide in a minute how hard to respond. PagerDuty’s published incident response documentation explains the convention in software: operational issues are classified at numbered severity levels, with the lower numbers more urgent, and anything above a set threshold is treated as a major incident with a more intensive response1.
For data, severity should be set by the consequence of the data being wrong or late, not by the size of the technical fault. A one-line bug that corrupts a regulatory return is severe. A failed cluster that delays an internal exploratory data set is not. The scale below is DII’s suggested adaptation.
A four-level severity scale for data incidents
| Level | Consequence | Response |
|---|---|---|
| 1. Critical | Wrong or missing data has reached, or will imminently reach, an external party, a regulator or a board-level decision | Declare immediately; named incident lead; senior stakeholders told at once; written review required |
| 2. High | Data that important internal reporting or operations depend on is wrong or unavailable now | Declare; named incident lead; affected users told promptly; written review required |
| 3. Moderate | A defined group of users is affected; a workaround exists; no decision has yet relied on the fault | Owning team fixes in working hours; users informed; logged |
| 4. Low | A fault with no current user impact | Logged and scheduled |
Two rules from PagerDuty’s guidance transfer without change. The first is to round up: “If you are unsure which level an incident is (e.g. not sure if SEV-2 or SEV-1), treat it as the higher one”1. The second is to defer the argument: “During an incident is not the time to discuss or litigate severities”1.
Knowing which data sits behind level one and level two has to be worked out beforehand. An organisation that has identified its most important data, as described in starting with critical data elements, can grade an incident quickly. One that has not will be debating importance while the incident runs.
Declare early, and name one person
Google’s Site Reliability Engineering book bases its incident practice on the Incident Command System used by emergency services, and stresses clear roles2. The central one is the incident commander, who “holds the high-level state about the incident”2. A data team does not need the full structure. It needs the principle: one named person coordinates, and that person is not also the one fixing the fault.
The same chapter gives a rule that would have prevented the duplicated reruns in the opening example: “The operations team should be the only group modifying the system during an incident”2. In a data incident, uncoordinated reruns and manual corrections are a real hazard. They can compound the damage and destroy the evidence of what went wrong.
On when to declare, the book’s advice is to err early: “It is better to declare an incident early and then find a simple fix and close out the incident than to have to spin up the incident management framework hours into a burgeoning problem”2. It suggests declaring if any of three questions is answered yes, of which two apply directly to data teams: “Do you need to involve a second team in fixing the problem?” and “Is the issue unsolved even after an hour’s concentrated analysis?”2.
Triage: contain first, diagnose last
Once an incident is declared, the lead works through a short sequence. The order is DII’s, and it puts containment before diagnosis.
- What is wrong, as the user sees it? Which data set, which period, wrong in what way.
- Is it still getting worse? If bad data is still being loaded or published, stop that first. Pause the pipeline, or hold the publication step. A pipeline designed to separate building from publishing, as described in what makes a data pipeline reliable, makes this a routine action.
- What is downstream? Use lineage to list every table, report, extract and feed built from the affected data. This sets the true scope.
- Who has used it? Identify the people and processes that consumed the data during the incident, and what they did with it.
- What is the cause? Only now. Diagnosis matters, and it comes after the spread has been stopped and the scope is known.
Throughout, the lead keeps a running record. The book calls this the commander’s “most important responsibility”: to “keep a living incident document”2. A shared page with a timeline of what was observed, decided and done serves the response as it happens and becomes the raw material of the review.
Communicate before you are asked
Communication is where trust is lost during an incident, and where it is cheapest to keep. Users forgive wrong data that they were told about. They do not forgive discovering it themselves.
DII’s suggested minimum is a message at each of these points.
- On declaring: what is affected, since when, what users should avoid relying on, and when the next update will come. Send this before the cause is known.
- At intervals: progress, and any change to scope. A promised update should be sent even when there is nothing new.
- On resolution: what was wrong, what has been corrected, which periods were affected and what anyone who used the data during the incident should now do.
The last element is the one specific to data. If figures were used in a decision or sent outside the organisation, the people responsible need to be told directly, with the corrected figures, and not left to infer it from a general notice.
Recover, and verify the recovery
Recovery has three parts: fix the cause, repair the affected data, and rebuild everything downstream in dependency order. Repair is quick where pipelines are idempotent and raw inputs were kept, and slow and manual where they are not.
Verify before closing. Confirm that the corrected data is right, by a check independent of the fix, and that every downstream item on the list from triage has been rebuilt. An incident closed on the assumption that the rerun worked is how the same incident is declared twice.
Learn: the written review
The practice that separates teams that improve from teams that repeat is the written review, called a postmortem in software engineering. Google’s definition is complete: “A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring”3.
It works under conditions that the sources are clear about.
It is blameless. The same source is unequivocal: “Blameless postmortems are a tenet of SRE culture”3. The review asks what in the system allowed a reasonable person to make the error, and assumes those involved acted in good faith on what they knew. This is a practical position as much as an ethical one. People who expect blame conceal what happened, and the review then learns nothing.
Its trigger is fixed in advance. Google advises: “It is important to define postmortem criteria before an incident occurs so that everyone knows when a postmortem is necessary”3. On the scale above, every level one and level two incident gets a review. Left to discretion, the review is skipped exactly when the team is most tired.
It is prompt and produces owned actions. PagerDuty’s own process sets deadlines for the review meeting, stating that it “should be scheduled within 3 calendar days for a SEV-1 and 5 business days for a SEV-2”4. Those are one company’s internal figures, cited to show that a deadline exists, and each team should set its own. Each action from the review needs an owner and a date, and is tracked to completion.
For data incidents the review should ask two questions that a software review might not: how long was the data wrong before anyone knew, and what check would have caught it sooner? The answers feed directly into where to put data quality checks in a pipeline.
Keeping a log
Individual reviews teach about individual incidents. A log of all incidents, including the minor ones, shows patterns: the source that fails every month-end, the table that is always involved, the class of fault that no check catches. A simple log with the date, severity, data affected, time to detect, time to resolve, cause and actions is enough to see them.
Time to detect is the figure most worth watching in a data team, because of the silence described at the start. A falling time to detect means monitoring is improving. The log is also evidence when a team asks for investment in reliability, and a basis for the service commitments discussed in running a data platform as a product.
The Data Incident & Issue Log provides the log and a review structure with the severity scale. The Data Observability Checklist covers the monitoring that shortens detection.
Sources
- PagerDuty. Severity Levels (PagerDuty Incident Response documentation).
- Stribblehill, A. (Google; Site Reliability Engineering, edited by Beyer, B. et al.). Managing Incidents (chapter 14 of Site Reliability Engineering). 2017.
- Lunney, J. and Lueder, S. (Google; Site Reliability Engineering, edited by Beyer, B. et al.). Postmortem Culture: Learning from Failure (chapter 15 of Site Reliability Engineering). 2017.
- PagerDuty. Postmortem Process (PagerDuty Incident Response documentation).
This article is general professional information. It is not legal or regulatory advice.