How to Evaluate and Select a Data Platform
A platform selection goes well when the requirements are written before any vendor is seen, the criteria are weighted before any score is given, and the shortlist is tested on the organisation's own workloads. It goes badly when the decision is made in a demonstration and justified afterwards.
A data platform is among the longer-lived technology decisions an organisation makes. Data accumulates in it, pipelines are written for it, people learn it, and the cost of leaving rises every month. A choice made in a few weeks is lived with for years.
That is a reason for method, and not for a lengthy procurement. This article sets out a selection process in six steps, the criteria that deserve weight, how to run a proof of concept that tests something, and how to think about lock-in. It is written for the people making or advising on the decision. It compares no products and recommends none.
Decide whether a selection is needed
The first decision is whether to select anything. Three questions are worth answering honestly.
What problem is the new platform to solve? Slow queries, rising cost, a capability the current platform lacks and an expiring contract are different problems with different answers. Some are solved by using the existing platform differently.
Is the architecture settled? A platform implements architectural decisions. If the organisation has not decided between the storage patterns compared in data warehouse, data lake or lakehouse, a product selection will make that decision implicitly, which is the wrong way round.
Is this a real choice? If the organisation is committed to one cloud provider and the outcome is in practice already decided, a full comparative exercise is theatre. A shorter confirmation that the default meets the requirements, with the gaps recorded, is more honest and more useful.
The UK government’s guidance on choosing technology frames the purpose of any such decision. It says the most important thing is to make choices that allow you to “change your mind at a later stage” and to “adapt your technology as your understanding of how to meet user needs changes”1. A selection process exists to make a choice the organisation can live with and, if necessary, leave.
What general technology guidance covers, and what it leaves out
Much of the published guidance on choosing technology is general. It was written for any system, and several of the sources used here were written for public bodies. Its principles carry over: keep the ability to change course, count the whole cost, prefer open interfaces, test before committing. The United States government’s Digital Services Playbook makes the same points as the UK guidance from a different administration, saying that technology choices “should seek to avoid vendor lock-in” and asking teams to “Consider open source software solutions at every layer of the stack”2.
What general guidance does not cover is the part that is particular to a data platform. Four things set the decision apart, and they are DII’s analysis.
- Data accumulates. An application can be replaced with its data exported. A data platform is where the data lives, and the volume held makes leaving progressively harder.
- The format is part of the product. How the platform stores data decides whether other engines can read it. The authors of the paper that introduced the lakehouse design put the concern strongly: “commercial warehouses lock data into proprietary formats that increase the cost of migrating data or workloads to other systems”3. The authors are affiliated with a vendor of the alternative they propose, and the claim should be read as an advocate’s. The question it raises is still the right one to ask of any candidate.
- Fit depends on the workload mix. Reporting queries, heavy transformation, data science and many concurrent users stress a platform in different ways. A product that suits one mix can be poor at another.
- Price follows behaviour. On consumption pricing the cost of a platform is set by how its users write and schedule work, which a feature comparison cannot show.
The selection method below is general in its discipline and specific in what it tests, for those reasons. The six steps and the criteria are DII’s own. No independent, published method for selecting a data platform was found and read for this article, and the sources cited support the principles behind the steps and not the steps themselves.
A six-step selection
Six steps in selecting a data platform
| Step | What is produced | The discipline |
|---|---|---|
| 1. Requirements | A statement of what the platform must do, by capability and by workload | Written before any vendor contact |
| 2. Criteria and weights | A short list of weighted criteria and any mandatory conditions | Agreed before any scoring |
| 3. Longlist to shortlist | Two or three candidates that meet the mandatory conditions | Eliminate on mandatory conditions only |
| 4. Proof of concept | Measured results on the organisation’s own workloads | Same tests, same data, every candidate |
| 5. Cost and commercial terms | A cost model over several years, with exit terms | Modelled on expected use, not list price |
| 6. Decision record | The choice, the reasons, what was rejected and why | The accountable people write it |
1. Requirements
Write down what the platform must do, organised by capability and by the workloads it will carry. The capabilities described in what a modern data platform is give the structure. For each, state what is needed, by whom, at what scale and with what constraints.
The Technology Code of Practice, the UK government’s standard for technology decisions, puts this first of its points, as “Define user needs”4. The users of a data platform are the engineers, analysts and applications that will work on it, and their needs are specific: the query patterns analysts run, the latency a data science team requires, the tools already in use that must connect.
Separate mandatory conditions from preferences. A mandatory condition is one whose failure eliminates a candidate: data residency in a required region, a necessary certification, integration with the identity system. There should be few. The Data Platform Requirements Template is organised to capture both.
2. Criteria and weights
Agree the criteria and their relative weight before looking at candidates. This is the single most important discipline in the process. Weights set after the demonstrations will be set, consciously or not, to favour the candidate the team already prefers.
DII suggests seven criteria. The weights are for each organisation to set.
- Fit to workloads. How well it does what the organisation’s actual workloads need.
- Total cost. Over several years, at expected usage, including the people needed to run it.
- Operability. The effort and skill required to run, secure and upgrade it.
- Security and compliance. Whether it meets the organisation’s obligations, with evidence.
- Openness and exit. How readily data and logic could be moved elsewhere.
- Ecosystem and skills. Whether the tools the organisation uses connect to it, and whether people who know it can be hired.
- Supplier viability and direction. Whether the supplier and the product are likely to remain suitable.
3. Longlist to shortlist
Reduce the field using the mandatory conditions alone. Do not score at this stage. Two or three candidates are enough for a proof of concept. More than that divides the team’s attention and extends the exercise without improving the decision.
4. Proof of concept
A demonstration shows what a product can do in the vendor’s hands. A proof of concept shows what it does in yours. The government guidance recommends the general approach to uncertainty: to “start with prototypes”1.
Four rules keep a proof of concept honest.
- Use your own workloads. Select a small number that represent the range: a heavy transformation, a query pattern with many concurrent users, an awkward ingestion. The vendor’s sample data proves nothing about your case.
- Define success beforehand. State the measures and thresholds for each test before running it.
- Run the same tests on every candidate, with the same data, and have your own people run them.
- Test the unglamorous parts. Setting up access control, deploying a change, recovering from a failed job and finding out what a workload cost. These decide what the platform is like to live with.
Limit the duration. A proof of concept that runs for months has become an implementation of the favoured candidate.
5. Cost and commercial terms
Model cost on the organisation’s expected use. Consumption-priced platforms make list price a poor guide, because cost depends on how workloads are written and run. Use the measurements from the proof of concept, scale them to expected volumes, and model growth. Include the cost of migration and of running the old and new platforms in parallel, covered in planning a data platform migration.
The government guidance asks teams to “minimise the total cost of ownership, including reducing the chance of getting locked in to long contracts for specific tools and providers”1. Commercial terms belong in the evaluation. Commitment length, the treatment of unused commitment, price protection and the terms on which data can be extracted at exit all affect the real cost.
6. Decision record
Write down the decision: what was chosen, the criteria and weights, the results, the options rejected and the reasons, and the risks accepted. The people who inherit the platform will want to know why. The form for recording a decision of this kind is described in architecture decision records for data teams.
Thinking clearly about lock-in
Lock-in is a criterion easily argued from principle and seldom analysed. Every platform creates dependence. The useful questions are how much, in what, and whether it is worth it.
UK government guidance on cloud lock-in takes a notably balanced position: “Typically, lock-in is considered something organisations should avoid. However, in the cloud the advantages of being locked-in can often outweigh the disadvantages”5. A provider’s proprietary services may be better and cheaper to operate than a portable alternative. Avoiding them entirely means forgoing that value and building more yourself.
The same guidance sets the aim as a balance: “You must aim to maximise both the value and portability of the services where possible”5. And it accepts the trade explicitly: “You may choose to accept a lower degree of portability if a service offers good value for money”5.
For a data platform, dependence arises in four places, and they differ in how hard they are to undo.
- Data. Whether data is held in open formats that other engines can read, and what it costs to move out.
- Logic. How much transformation code uses features specific to the platform.
- Integrations. How many tools and pipelines are connected through platform-specific interfaces.
- Skills. How much of the team’s knowledge is particular to the product.
Data in open formats and logic in standard query language are the two choices that most reduce the cost of a later exit. The guidance also advises planning for the case where portability is low: “you should consider how you would change providers in future and under what circumstance it would compel you to do so”5. An exit plan written at selection, even a page, is evidence that the dependence was chosen knowingly.
Ways the process goes wrong
Five patterns undermine selections. They are DII’s cautions.
Deciding at the demonstration. A polished demonstration creates a preference that the rest of the process then rationalises. Fixing criteria and weights first is the defence.
Evaluating features, not workloads. A long matrix of features, each ticked, shows that every serious product does most things. It does not show which one suits the work.
Ignoring the cost of operation. The licence or consumption charge is visible. The engineers needed to run the platform are a larger cost and appear in a different budget.
Leaving out the people who will use it. A platform chosen by a procurement exercise without the engineers and analysts who will work on it meets resistance on arrival.
Treating selection as the end. The choice starts a migration and a period of operation that will cost more than the selection. Who will run the platform, and on what service model, should be known before the contract is signed; running a data platform as a product sets out the questions.
After the decision
A selection is finished when three things exist: a decision record, a cost model that will be compared with actual spend, and a plan for migration and operation. The Data Platform Evaluation Guide provides the criteria, a weighting method and a scorecard for the comparison, with space for the proof-of-concept results.
Sources
- Central Digital and Data Office (UK Government). Choosing technology: an introduction (Service Manual). Updated 15 October 2024.
- United States Digital Service. Digital Services Playbook.
- Armbrust, M., Ghodsi, A., Xin, R. and Zaharia, M. Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. 11th Conference on Innovative Data Systems Research (CIDR 2021). January 2021.
- Central Digital and Data Office (UK Government). The Technology Code of Practice. Updated 7 July 2025.
- Central Digital and Data Office (UK Government). Managing technical lock-in in the cloud. Updated 3 September 2026.
This article is general professional information. It is not legal or regulatory advice.