Modern Data Architecture: The Layers, Patterns and Decisions That Matter
A data architecture is easier to describe by the decisions it fixes than by the products it runs on. The named patterns, from warehouse to mesh, are different answers to the same short list of questions.
A data architecture is the set of structural decisions about how an organisation’s data is brought in, stored, shaped, served, described, protected and owned. Products implement those decisions. They do not make them, and an architecture described as a list of products says very little about why it has the shape it has.
That distinction matters most at the moment an architecture has to be reviewed, defended or changed. A diagram of vendor logos can be redrawn in an afternoon. The decisions underneath it, such as where data is given its structure, or which team answers for a dataset’s meaning, take years to reverse. This article sets out those decisions, shows a vendor-neutral way to draw the layers, and reads the familiar named patterns as answers to the same questions. It is the starting point for the rest of DII’s coverage of the subject.
Architecture is the part that is expensive to change
Not every technical choice is architectural. The useful test is cost of reversal. Choosing a dashboard tool for one team can be undone in a quarter. Deciding that all analytical data will be modelled centrally before anyone may use it shapes hiring, funding and the pace of every later project.
Two properties mark a decision as architectural. It constrains many later decisions, and it is costly to revisit once work has been built on it. Everything in this article is about decisions of that kind. Choices below that line belong to engineering teams and should be left to them.
The definition is DII’s. It is consistent with how the sources cited here treat the subject, and none of them states it in these words.
Seven decisions every data architecture fixes
Whatever it is called, a data architecture has taken a position on seven questions. Some organisations take these positions deliberately. Others inherit them from the first platform they bought. Either way the positions exist, and a review can find them.
The seven decisions, the question each answers and what it constrains
| Decision | The question | What it constrains afterwards |
|---|---|---|
| Ingestion | How does data arrive: in batches, as a continuous stream, or both? | How fresh anything downstream can be, and how many code paths must agree |
| Storage | In what form is data kept, and is storage separate from the engines that query it? | Cost behaviour, which tools can read the data, and how hard it is to leave |
| Modelling | When is structure imposed: as data is written, or when it is read? | Who carries the cost of making data usable, and how quickly a new source can be added |
| Serving | How do people and systems get data out: queries, files, interfaces, events? | Which uses are well supported, including reporting, data science and applications |
| Metadata | How is data described, found and traced? | Whether anyone can discover a dataset, trust it, or follow it back to its source |
| Security | Where are access and protection rules enforced? | Whether a rule holds across every route to the data or only the main one |
| Ownership boundaries | Which team answers for which data, and where does a central team’s responsibility stop? | Where bottlenecks form, and who can change what without asking |
The first five are technical in appearance and the last two organisational, but they are not independent. A decision to impose structure late moves work from the team that loads data to the teams that use it. A decision to let many engines read the same files moves the enforcement of security rules out of any one engine. Reading an architecture well means following these consequences from one row of the table to the next.
Structure early or structure late
The modelling decision has the longest history and still divides the main patterns. Microsoft’s architecture guidance states the contrast compactly. A data lake approach means that “Data lakes store all data types in their original, untransformed state and apply transformation only when the data is needed”, which is known as schema-on-read, while a data warehouse “enforces structure and applies transformations as it ingests data”, known as schema-on-write1.
Neither is free. Structure imposed early costs time before any data can be used and protects every later reader. Structure imposed late lets data land quickly and leaves each reader to work out what it means.
One path or two
The ingestion decision has its own well-documented trap. The Lambda Architecture, proposed for systems that need both complete historical results and low-latency ones, runs a batch system and a stream system side by side. Jay Kreps, who built streaming infrastructure at LinkedIn, described the consequence in 2014: “You implement your transformation logic twice, once in the batch system and once in the stream processing system”2. His objection was practical. In his words, “maintaining code that needs to produce the same result in two complex distributed systems is exactly as painful as it seems like it would be”2.
The lesson generalises beyond the two technologies he was writing about. Any architecture that computes the same business result along two routes has to keep the routes in agreement, and that upkeep is a permanent cost that belongs in the decision.
The layers, drawn without a vendor
An architecture is presented as a layer diagram, and one published by a vendor is a diagram of that vendor’s products. A neutral one exists. The US National Institute of Standards and Technology published a reference architecture for large-scale data systems that it describes as a “vendor-neutral, technology- and infrastructure-agnostic conceptual model”3.
The model has five roles, which it presents as “technical roles that exist in every Big Data system”: a system orchestrator, a data provider, an application provider, a framework provider and a data consumer3. Around them sit two cross-cutting concerns, management and security and privacy, which the document calls fabrics and which “provide services and functionality to the five main roles”3. Inside the application provider it names five activities that will be familiar from any pipeline diagram: “collection, preparation/curation, analytics, visualization, and access”3.
The value of the model for a practitioner is as a checklist of what must be present. Data has to come from somewhere and go to someone. Something has to collect, prepare, analyse and expose it. Something has to supply storage and compute. Something has to coordinate the whole, and management and security have to reach every part. A diagram that omits one of these has not removed the need. It has left the responsibility unassigned.
A second, simpler layering is used inside lake-based systems. Microsoft’s guidance describes raw, cleansed and curated layers, often labelled bronze, silver and gold, and says that this arrangement, known as the medallion architecture, “improves data quality, governance, and performance”1. It is a layering of data by degree of refinement, where the NIST model is a layering of responsibilities. The two answer different questions and can be used together.
Named patterns are answers to the same questions
The names that dominate architecture discussions are not rival theories of data. Each is a set of answers to the seven decisions, with one or two answers emphasised.
The data warehouse answers the modelling question first: structure is imposed as data is written. The authors of the 2021 paper that named the lakehouse describe warehouse data as “written with schema-on-write, which ensured that the data model was optimized for downstream BI consumption”4. Serving is by SQL to reporting tools. Storage and compute were originally bought together.
The data lake answers the storage question first: keep everything, cheaply, in open file formats, and decide on structure later. The same paper traces how organisations then ran both, and what it cost them: “data is first ETLed into lakes, and then again ELTed into warehouses, creating complexity, delays, and new failure modes”4.
The lakehouse is an attempt to stop paying for two systems. It keeps the lake’s storage decision and adds a metadata layer that supplies what warehouses were valued for, which the paper lists as “ACID transactions, data versioning, auditing, indexing, caching, and query optimization”4. Its authors are affiliated with Databricks, a company that sells such a system, and the paper should be read as an argument as well as a description. The comparison is taken up criterion by criterion in data warehouse, data lake or lakehouse.
Data mesh leaves the first five decisions largely open and answers the last one. Its originator, Zhamak Dehghani, grounds it in four principles: “domain-oriented decentralized data ownership and architecture, data as a product, self-serve data infrastructure as a platform, and federated computational governance”5. It is a position on ownership boundaries with consequences for everything else. Whether that position suits a given organisation, and how it differs from a data fabric, is the subject of data mesh vs data fabric.
Seen this way, the patterns are not mutually exclusive. An organisation can run a lakehouse and organise ownership by domain. It can keep a warehouse for regulated reporting beside a lake for data science. What it cannot do is avoid answering the seven questions, and adopting a named pattern does not answer the ones the pattern is silent on.
Operational and analytical data are different problems
One boundary runs underneath all of the above. Dehghani’s description is a useful statement of it: “Operational data sits in databases behind business capabilities served with microservices, has a transactional nature, keeps the current state and serves the needs of the applications running the business”5. Analytical data, by contrast, “is a temporal and aggregated view of the facts of the business over time”5.
The patterns discussed here are all architectures for the analytical side. Each depends on data crossing that boundary reliably, and the crossing is where the pipelines and their failures are. An architecture document that describes only the analytical estate, and treats the source systems as a single box on the left, has left out the part on which the rest depends. Mapping those crossings is covered in how to map data flows across systems.
Architecture is not the platform
A data platform is the set of products and services that implements an architecture. The two are confused because platform vendors publish reference architectures in which every box is one of their products.
The practical test is substitution. If a component could be replaced by a competitor’s without changing who owns what, where structure is imposed or how data is served, the choice was a platform decision. If replacing it would change one of the seven answers, it was architectural and was presented as a purchase. What a platform has to provide, and where its boundaries lie, is the subject of what a modern data platform is.
The same test applies to governance. An architecture decides where rules can be enforced. It does not decide what the rules are or who sets them, which is the business of data governance and what a working programme contains.
Reading an architecture you did not design
A practitioner who inherits an architecture has to work out what it decided. Five questions recover the positions quickly.
- Where does a new source first land, and how long before anyone can query it?
- At what point is data given agreed structure and meaning, and who does that work?
- How many different routes produce the figures the executive team sees?
- If a dataset is wrong, who is told, and who has the authority to fix it at source?
- Which components could be replaced without changing any of the answers above?
The answers are the architecture, whatever the diagram says. Where they were never written down, the first task is to record them, and architecture decision records for data teams describes a practice for doing so. Where the answers are known and suspected to be wrong, running a data architecture review that changes something covers how to test them.
The Architecture Review Checklist sets out review questions against each of these areas. The Data Architecture Assessment is the fuller instrument for scoring the current state.
Sources
- Microsoft. What is a data lake? (Azure Architecture Center, Microsoft Learn). Updated 21 April 2026.
- Kreps, J. Questioning the Lambda Architecture (O’Reilly Radar). 2 July 2014.
- NIST Big Data Public Working Group (National Institute of Standards and Technology, United States). NIST Big Data Interoperability Framework: Volume 6, Reference Architecture, Version 3 (NIST SP 1500-6r2). October 2019.
- Armbrust, M., Ghodsi, A., Xin, R. and Zaharia, M. Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. 11th Conference on Innovative Data Systems Research (CIDR 2021). January 2021.
- Dehghani, Z. Data Mesh Principles and Logical Architecture (martinfowler.com). 3 December 2020.
This article is general professional information. It is not legal or regulatory advice.