Running a Data Architecture Review That Changes Something
A review that ends in a long list of findings has done the easy half. The half that changes anything is deciding in advance which decision the review serves, and leaving the room with an owner and a date against each thing that matters.
An architecture review is commissioned, a team spends weeks on it, and the output is a long document of findings sorted by severity. It is presented, thanked for and filed. The next review finds the same things and adds a few.
The failure need not be in the analysis. It can lie in the framing before the review and the handling after it. A review framed as a general health check has no client, so its findings have no owner. This article sets out a way of running a data architecture review so that it leads to funded action: scope it to a decision, gather evidence in place of opinion, sort what is found, rank it by risk and by the cost of waiting, and present it so that someone is answerable. The method borrows from two published sources, an evaluation method from Carnegie Mellon’s Software Engineering Institute and a cloud vendor’s review process, and adapts them to data. The adaptation is DII’s.
Why reviews produce lists
A list of findings is what a review produces by default, because finding things is the part reviewers control. Three conditions make the list the end of the matter.
The review has no decision attached. Nobody was waiting for the answer in order to choose something, so no one is obliged to act on it.
The findings are unranked in any way a budget holder can use. Severity ratings describe how bad a thing is. They do not say what happens if it waits a year, or what it would cost to fix.
The findings have no owner. A statement that lineage is incomplete is true and addressed to nobody.
Amazon Web Services, describing how it reviews architectures, puts the intended result in one sentence: “The outcome of the review is a set of actions that should improve the experience of a customer using the workload”1. The operative word is actions. Everything below is about arranging the review so that actions are what comes out.
Scope the review to a decision someone has to make
The first question is not what to review. It is who needs to decide what, and by when.
A data architecture review earns its cost when it is attached to one of a small number of decisions: whether to migrate or extend a platform, whether the estate can support a planned use such as a regulatory return or a machine learning programme, where next year’s engineering budget should go, or whether a recurring class of incident has an architectural cause. Each has a decider and a deadline. The decider is the review’s client.
The Software Engineering Institute’s Architecture Tradeoff Analysis Method builds this into its sequence. Before any architecture is presented, the method has a step in which the business drivers are presented2. The architecture is then examined against what the organisation needs from it and not against an abstract standard of quality.
Scoping to a decision also settles the boundary. A review to inform a platform migration looks hard at what is tied to the current platform and lightly at dashboard design. Without the decision there is no principled way to leave anything out, and a review that leaves nothing out finishes late and says little. The layers and decisions described in modern data architecture: the layers, patterns and decisions that matter give a frame for stating what is in scope.
Timing follows from the same logic. AWS recommends reviewing “early on in the design phase to avoid one-way doors that are difficult to change”1. A review after the contract is signed can only describe consequences.
Collect evidence before opinion
Left alone, a review collects opinions: the architect’s view of the warehouse, the analysts’ view of the architect. Opinions are useful for knowing where to look. They are not findings.
A finding rests on something another person could check. For a data architecture, five sources of evidence do most of the work.
Evidence to gather before forming a view
| Evidence | What it shows | What its absence shows |
|---|---|---|
| Data flow map | What depends on what; where copies and manual steps are | Nobody can state the impact of a change |
| Decision records | What was chosen, why, and what cost was accepted | Current structure cannot be told apart from accident |
| Incident and failure log | Where the architecture breaks in practice, and how often | Failures are being absorbed without being counted |
| Cost and usage reports | Which stores and pipelines consume money, and which are unused | Spend cannot be related to value |
| Access and ownership register | Who can reach what, and who answers for it | Control depends on individual memory |
The table is DII’s. The third column matters as much as the second. When a piece of evidence does not exist, that is a finding, and it can matter more than anything the evidence would have shown.
Two of these have their own treatment. Building and maintaining the map is covered in how to map data flows across systems and keep the map current. The records are the subject of architecture decision records for data teams; Microsoft’s guidance gives the reason they come first in a review: “Your architecture is the accumulation of its decisions”3. Failure patterns in pipelines, which the incident log will surface, are examined in what makes a data pipeline reliable.
Conversations still matter, and the AWS process is a good model for their tone. It asks for “a blame-free approach that encourages diving deep” and describes the review as “a conversation and not an audit”1. It also reports something reviewers should expect: “Often, we find that reviews are the first time that a team truly understands what they have implemented”1. A team that is learning about its own system in the review will not volunteer what it knows if it expects to be blamed for it.
Sort what you find into three kinds
The default output is a single list ordered by severity. The Software Engineering Institute’s method offers a more useful sorting, which separates three things a review discovers.
Risks. The method defines them briefly: “Risks are potentially problematic architectural decisions”2. Its notion of a decision includes one that has not been taken. An undefined rule for how business keys are assigned is a risk in this sense, though nobody decided anything.
Sound decisions that depend on an assumption. The method calls these non-risks: “Non-risks are good decisions that rely on assumptions that are frequently implicit in the architecture”2. Its instruction covers both categories: “Both should be understood and explicitly recorded”2. A nightly batch load is a sound decision while the business accepts day-old figures. Writing the assumption down tells the organisation what would turn a sound decision into a risk.
Trade-off points. These are the places where one property of the design affects two things the organisation wants, in opposite directions. The method’s wording is “a property that affects more than one attribute and is a sensitivity point for more than one attribute”, and its assessment of their importance is unambiguous: “Tradeoff points are the most critical decisions that one can make in an architecture”2. In a data estate, how long raw data is retained is one: longer retention helps reprocessing and audit and raises cost and exposure.
The method was designed for software systems and its quality attributes are performance, availability, security and modifiability. For data, DII would add correctness, timeliness and traceability. The sorting carries over unchanged.
Recording the second category is what makes a review credible to the people whose work is being reviewed. A report that lists only faults reads as an indictment. One that also says which decisions were sound, and why, shows that the reviewers understood the system.
Rank by risk and by the cost of waiting
A sorted list is still a list. To be acted on, the risks have to be ordered in terms a budget holder uses. Severity alone does not do that. AWS’s guidance leaves the ordering to the organisation and says on what basis: after a review, “you should have a list of issues that you can prioritize based on your business context”1.
DII’s suggestion for making that concrete is to answer four questions for each risk. They are judgement, not a formula.
- What happens, to whom, if it materialises? A named consequence: a late regulatory return, a wrong figure in the board pack, a week of outage.
- How likely is that in the planning period? The incident log is the best guide. Something that has happened twice is likelier than something that could in theory happen.
- What does waiting cost? Some risks are stable. Others grow: each new pipeline built on an unreliable identifier adds to the eventual bill. A growing cost is the strongest argument for acting now.
- What does fixing it cost, and does it unblock anything else? A cheap fix that removes an obstacle to three planned projects outranks an expensive one that stands alone.
Three or four risks will separate from the rest. Those are the review’s recommendations. The remainder go in an appendix, recorded and deliberately not acted on this year, with the reason.
Where the decision behind the review is a migration, the ranking feeds directly into its phasing: the risks that must be retired before cutover, and those that the move itself will remove. Planning a data platform migration takes that up.
Present each finding with an owner and a date
The presentation is where a review becomes action or does not. Four elements belong against each recommended item.
- The finding, with its evidence. One sentence and a pointer to what supports it.
- The consequence, in the client’s terms. What it puts at risk in the decision the review was scoped to.
- A proposed owner. A named role with the authority and budget to act, agreed with that person before the meeting.
- A date for a decision. Not necessarily a date for the fix. A date by which the owner will accept, fund, defer or reject the recommendation.
The fourth element is the easiest to leave out. Deferral and rejection are legitimate outcomes. An owner who says that a risk is understood and accepted for a year has made a decision that can be recorded and revisited. What a review must not leave behind is a finding nobody has answered.
Acceptance is not the end either. The UK Government Digital Service makes the point about decisions in general, in its engineering standards: “Consider using your issue tracker to make sure that decisions you’ve made are followed through and put into effect”4. Review actions belong in the same backlog as other work, where they compete for capacity in the open and can be seen not to have moved.
When the review is unwelcome
A review examines other people’s work, and resistance is an understandable response. The AWS guidance lists objections it meets and its answers to them, including the claim of being too busy before a launch and one that goes to the heart of this article: “We don’t have time to do anything with the results!”1.
That objection deserves a better answer than reassurance. It is a reasonable prediction based on previous reviews. The answer is in the design: a review scoped to a decision the team itself is waiting on, short enough not to compete with delivery, and committed in advance to a handful of recommendations.
On length, the same guidance is specific that a review “should be a lightweight process (hours not days)”1. That describes a review of one workload against a set of standard questions. A review of a whole data estate ahead of a migration is a larger exercise, and the principle still applies: the smallest review that answers the client’s question is the right size, and one that is small enough can be repeated.
Keep it small enough to repeat
A single large review ages quickly. A modest one, repeated when something significant changes, keeps its findings current and makes each round cheaper than the last, because the evidence is already assembled.
The Architecture Review Checklist provides the standard questions, grouped by the areas a data architecture has to cover, and is free to use. Where a fuller scored baseline is needed, for example ahead of a multi-year investment, the Data Architecture Assessment is the instrument.
The measure of a review is simple and can be checked a quarter later. Count the recommendations that have an owner’s recorded decision against them. If that number is close to the number of recommendations made, the review changed something. The length of the findings list was never the point.
Sources
- Amazon Web Services. The review process (AWS Well-Architected Framework).
- Kazman, R., Klein, M. and Clements, P. (Software Engineering Institute, Carnegie Mellon University). ATAM: Method for Architecture Evaluation (Technical Report CMU/SEI-2000-TR-004). August 2000.
- Microsoft. Maintain an architecture decision record (ADR) (Azure Well-Architected Framework, Microsoft Learn). Updated 13 April 2026.
- Government Digital Service (UK). Documenting architecture decisions (The GDS Way). Updated 5 March 2026.
This article is general professional information. It is not legal or regulatory advice.