What a Modern Data Platform Is: Capabilities, Components and Boundaries
A data platform is best described by what it lets people do: bring data in, keep it, shape it and serve it, safely and dependably. Products come and go beneath that description. The capabilities, and the boundary between the platform and the teams that use it, are what an organisation has to decide.
An organisation’s data platform is commonly described by naming its products: the cloud warehouse, the lake, the orchestration tool, the BI server. Each is part of the answer, and the list as a whole explains very little. It does not say what the platform is for, what is missing, or who is responsible when something falls between two products.
A more useful description starts from capability. This article defines a data platform, follows data through it to show what the platform has to be able to do, relates those capabilities to the components that implement them, and draws the boundaries between a platform and the things it is confused with. It takes no view on which products to buy.
A working definition
A data platform is the shared set of technology and services through which an organisation collects, stores, prepares and provides data for use, together with the controls that make that safe and the operations that keep it running.
Two parts of that definition carry weight. The first is shared. A platform serves many teams and uses. A single team’s database is a system. The second is services. A platform is something people use to get their own work done without building the underlying machinery themselves.
The Cloud Native Computing Foundation’s white paper on platforms, written about software delivery, gives a definition that applies equally to data: “A platform for cloud-native computing is an integrated collection of capabilities defined and presented according to the needs of the platform’s users”1. The two ideas in that sentence, a collection of capabilities and the needs of users, organise the rest of this article.
Why describe it by capability
Products change faster than needs. The need to store data durably and query it has been constant for decades, while the products that do it have been replaced several times. A description in terms of capability stays true through those replacements. It also exposes gaps. An inventory of products shows what has been bought. A list of capabilities shows what has not been provided.
Reference architectures take this approach. The NIST Big Data Reference Architecture describes what is done to data across its life cycle as a set of activities, “collection, preparation/curation, analytics, visualization, and access”2, and it treats security, privacy and management as fabrics that run through every component and are not layers of their own2. Amazon’s guidance for analytics workloads makes a related point about integration: “Modern data architecture integrates a data lake, a data warehouse, and other purpose-built data stores while enabling unified governance and seamless data movement”3.
The arrangement below is DII’s, informed by those sources. Some capabilities follow the path of the data from arrival to use. The others apply at every point along that path. The table is a reference, and the explanation is in the two sections after it.
The capabilities of a data platform
| Capability | What it provides | Typical components |
|---|---|---|
| Ingest | Bringing data in from sources, in batches or continuously | Connectors, change data capture, event streams, file transfer |
| Store | Keeping data durably, at reasonable cost, in usable formats | Object storage, warehouse storage, open table formats |
| Transform | Turning raw data into modelled, usable data | Query engines, processing frameworks, transformation tooling, orchestration |
| Serve | Making data available to people and systems | SQL endpoints, semantic layer, BI tools, APIs, data sharing |
| Govern | Knowing what data exists, what it means and who is accountable | Catalogue, glossary, lineage, classification |
| Secure | Controlling and evidencing who can do what | Identity, access control, encryption, masking, audit logs |
| Observe | Knowing whether data and pipelines are healthy | Monitoring, data quality checks, alerting |
| Operate | Running, changing, supporting and paying for the platform | Environments, deployment automation, cost management, support process |
Following the data: ingest, store, transform, serve
Ingest. Data has to arrive from operational systems, external providers and devices. The capability includes handling different arrival patterns, coping with changes at the source and recording what arrived and when. Its weakness shows as each new source being a bespoke project.
Store. The platform keeps data durably and affordably, in forms that the other capabilities can use. The principal choices here, between warehouse, lake and lakehouse designs, are architectural, and are compared in modern data architecture: the layers, patterns and decisions that matter.
Transform. Raw data becomes useful through cleaning, joining and modelling. The capability covers the engines that do the work, the tooling in which logic is written and tested, and the orchestration that runs it in order. The properties that make this dependable are the subject of what makes a data pipeline reliable.
Serve. Data is of value when it reaches a person or a system that uses it. Serving includes query access for analysts, the definitions layer that keeps measures consistent, BI tools, programmatic interfaces for applications and, increasingly, access for machine learning and AI workloads. The particular demands of the last are examined in is your data platform ready for AI workloads?.
These follow the path of the data. Analysis is sometimes listed separately. Here it is treated as a use of the serve capability, since the platform’s job is to make analysis possible and the analysis itself belongs to its users.
The capabilities that run across the others
Govern. People need to find data, understand it and know who is accountable for it. The platform contributes the tooling: a catalogue, lineage, the means to record classification. It does not contribute the decisions. Who owns a data set and what a term means are settled by the organisation, and a catalogue with nothing agreed to put in it is empty.
Secure. Access must be controlled, sensitive data protected and both demonstrated to an auditor. This cannot be added at one layer. It applies at ingestion, in storage, in transformation and at every point of serving, which is why NIST draws it as a fabric around the whole architecture2.
Observe. The platform has to make it possible to know whether data arrived, whether it is complete and whether jobs succeeded. Without this, the first signal of a fault is a user’s complaint.
Operate. A platform is run, and running it is a capability with its own requirements: separate environments for development and production, automated and repeatable deployment, support when users are stuck, capacity management, and visibility of what everything costs. Cost in particular is a property of how the platform is operated day to day, discussed in controlling data platform costs.
The cross-cutting capabilities are the ones a platform grown by acquisition of products is likeliest to lack. Each product ingests, stores or transforms. Few purchases are made in order to operate.
Capabilities are not products
The mapping between capabilities and products is many to many. One product may provide several capabilities: a cloud warehouse stores, transforms and serves. One capability may need several products: serving may involve a query endpoint, a BI tool and an interface for applications.
Two consequences follow. An organisation can have every fashionable product and still lack a capability, and the cross-cutting ones are the likeliest gaps. And it can have two or three products providing the same capability, each adopted by a different team, with the cost and inconsistency that implies. Laying the products an organisation owns against the capabilities shows both conditions at once.
Amazon’s guidance warns of what happens when stores multiply without design: “This data movement can get complex and messy as the data grows in these data stores”3. The capability view is a way to see that complexity before adding to it.
Where the platform ends
Three boundaries are worth drawing clearly, because each is easily blurred.
A platform is not an architecture. Architecture is the set of decisions about structure: which patterns, which layers, how data flows. The platform is the running implementation of some of those decisions. A new platform does not correct an unsound architecture. It reproduces it at higher speed.
A platform is not a data product. The platform is what data products are built on. The sales data set, the customer model and the finance dashboard are built by teams using the platform, and they are owned by those teams. This boundary decides who is called when a number is wrong. If the pipeline engine failed, it is the platform team. If the logic in the pipeline is wrong, it is the team that wrote it.
A platform is not a team. It is a set of capabilities, and those need people to provide them, with a clear account of what they offer and to whom. The white paper cited earlier makes this the first attribute of a successful platform: “A platform exists to serve the requirements of its users and it should be designed and evolved based on those requirements, similar to any other software products”1. What that means for staffing, service and funding is the subject of running a data platform as a product.
What makes a platform modern
The word modern is used loosely, and it is better tied to properties than to a date or a product generation. The properties that distinguish current designs from their predecessors are these. The summary is DII’s.
- Elastic. Storage and processing scale separately and on demand, so capacity is not fixed by an annual purchase.
- Open at its interfaces. Data is held in formats, and reached through interfaces, that more than one tool can use. The UK government’s technology guidance gives the general advice: “If they’re available you should use open standards to manage the interfaces between parts of your system”4.
- Self-service. Users can obtain what they need without raising a request for each thing. The white paper is categorical: “A platform should be self-serviceable”1.
- Automated in its controls. Access, quality checks and deployment are applied by code and configuration, consistently, and not by manual procedure.
None of these requires a particular vendor, and a platform can have all of them on technology that is several years old. Equally, a recently purchased product delivers none of them by itself if it is operated through tickets and manual steps.
The same government guidance offers a test for any technology choice that is especially apt for platforms, which are long-lived: the most important thing is to make choices that allow you to “change your mind at a later stage”4.
Using the capability view
The capability view earns its place in practice.
Assessing what you have. For each capability, ask who uses it, what provides it, how well, and who is responsible. Gaps and duplicates become visible.
Stating what you need. Requirements written against capabilities describe outcomes and leave room for different solutions. Requirements written as product features presuppose the answer. The Data Platform Requirements Template is organised on this basis.
Comparing options. When a platform decision is to be made, capabilities give the criteria. The method is set out in how to evaluate and select a data platform, supported by the Data Platform Evaluation Guide. Where the decision is to move from one platform to another, the planning is covered in planning a data platform migration.
Sources
- CNCF TAG App Delivery (Cloud Native Computing Foundation). CNCF Platforms White Paper.
- NIST Big Data Public Working Group (National Institute of Standards and Technology, United States). NIST Big Data Interoperability Framework: Volume 6, Reference Architecture, Version 3 (NIST SP 1500-6r2). October 2019.
- Amazon Web Services. Modern data architecture (Data Analytics Lens, AWS Well-Architected Framework).
- Central Digital and Data Office (UK Government). Choosing technology: an introduction (Service Manual). Updated 15 October 2024.
This article is general professional information. It is not legal or regulatory advice.