Lakehouse Architecture

What it is

The paper that named the pattern, written by authors at a vendor, argues that keeping a data lake and a warehouse consistent is difficult and costly, and describes a lakehouse as adding ACID transactions, data versioning, auditing, indexing, caching and query optimisation. It also notes that lakehouses give up some aspects of data independence.

An independent review from a university and an industrial research group offers a definition: a lakehouse is an integrated data platform that leverages the same storage type and data format for reporting and OLAP, data mining and machine learning, as well as streaming workloads.

Why it matters

Keeping a lake and a warehouse consistent is costly, and the lakehouse removes a copy. But the label is used loosely: an independent review found existing definitions vague and often driven by marketing.

The same review tested six tools against its requirements and found that only three, all open table formats, satisfied them all. It dates from 2023 and assesses tools as they then were.

The problem it addresses

DII editorial view

Many organisations run a lake for data science and a warehouse for reporting, with pipelines copying data between them. The two drift apart and the copying is expensive to maintain. The lakehouse pattern serves both kinds of work from one copy; the difficulty is that the label is also applied to products that do not do this.

Business and organisational relevance

DII editorial view

The case is usually cost and simplicity: one platform and one copy in place of two. Because the word is used loosely, the commercial risk is buying something called a lakehouse that does not behave like one.

Practical use cases

• Replacing a separate lake and warehouse with one platform
• Serving reporting and machine learning from the same data
• Reducing the pipelines that copy data between a lake and a warehouse

Benefits and opportunities

• One copy of data to keep consistent
• Open storage formats beneath the platform

Risks, limitations and dependencies

• The label is applied to products that do not meet an independent definition
• Some aspects of data independence are given up
• DII has read no source that measures adoption

Prerequisites for adoption

DII editorial view

• A clear statement of the workloads the platform must serve
• Open storage formats, and a table format that supports transactions
• An architecture decision record, so the trade-offs are written down

Implementation considerations

• Ask what the supplier means. Test the claim against one storage type and one format for all workloads.
• Record the trade-offs. Write down the choice and what it gives up, as an architecture decision.
• Check the table format. The independent review found the capability in the table formats, not in the label.

The DII Radar assessment

Approved DII Radar assessment

DII assesses Lakehouse Architecture as High relevance, with direction Stable. Its strategic impact is assessed as High and its enterprise adoption as Scaling. It is in the catalogue and is not one of the ten entries on the current Radar.

DII relevance is our assessment of current significance and momentum, based on evidence, market development and adoption. Direction is the observed trajectory of relevance and adoption. Strategic impact is the potential or current significance to enterprise data capability, and enterprise adoption is the extent of current real-world enterprise adoption; together they set the position on the Radar. The Radar is not a prediction, vendor comparison or performance ranking.

Evidence basis: Armbrust et al., Lakehouse (CIDR 2021; vendor authors); Schneider et al., Assessing the Lakehouse (ICEIS 2023; University of Stuttgart and Bosch).

What to consider next

DII editorial view

• Ask what a supplier means by "lakehouse" and test it against one storage type and format for all workloads
• Record the choice and what it gives up
• Check the table format; the independent review found the capability there, not in the label