Open Table Formats

What it is

Apache Iceberg is the example DII has read: an open table format for huge analytic datasets, usable from Spark, Trino, PrestoDB, Flink, Hive and Impala. Table changes are atomic, and readers never see partial or uncommitted changes.

The format sits between storage and compute. Data stays in files in object storage; the format adds the table: its schema, its snapshots and the record of what changed.

Why it matters

An open table format separates the choice of storage from the choice of engine, which lowers switching cost. A major cloud provider now offers the format as a managed storage service.

On 3 December 2024 AWS announced Amazon S3 Tables, described as a cloud object store with built-in Apache Iceberg support. That is evidence that the format is available as a managed service. It is one vendor announcement, and DII has read no source showing how widely the format is used.

The problem it addresses

DII editorial view

When the table format belongs to one engine, adding a second engine or replacing the first means converting the data. That cost shapes every later platform decision. An open format lets several engines work on the same tables in place.

Business and organisational relevance

DII editorial view

The practical effect is on procurement. If tables are held in an open format, a second engine can be added, or the first replaced, without first converting the data.

Practical use cases

• Several engines working on the same tables without copies
• Keeping a route open to change the query engine later
• A common table layer beneath a lakehouse

Benefits and opportunities

• Lower switching cost between engines
• One copy of the data for several kinds of workload

Risks, limitations and dependencies

• The format is open but the catalogue around it may not be portable
• Engines differ in how fully they support a format
• Vendor performance claims are not evidence and are not repeated here

Prerequisites for adoption

DII editorial view

• Data held, or planned to be held, in object storage
• Clarity on which engines must read and write the tables
• A catalogue decision; the tables may be portable when the catalogue is not

Implementation considerations

• Ask what is native. When selecting or renewing a platform, ask which open table formats it reads and writes natively.
• Check the catalogue. Ask whether the catalogue is portable, not only the tables.
• Test with your engines. Confirm that each engine you rely on supports the features you need.

The DII Radar assessment

Approved DII Radar assessment

DII assesses Open Table Formats as High relevance, with direction Rising. Its strategic impact is assessed as High and its enterprise adoption as Emerging. It is one of the ten entries on the current Radar.

DII relevance is our assessment of current significance and momentum, based on evidence, market development and adoption. Direction is the observed trajectory of relevance and adoption. Strategic impact is the potential or current significance to enterprise data capability, and enterprise adoption is the extent of current real-world enterprise adoption; together they set the position on the Radar. The Radar is not a prediction, vendor comparison or performance ranking.

Evidence basis: Apache Iceberg documentation and table specification (Apache Software Foundation); AWS announcement of Amazon S3 Tables, 3 December 2024 (vendor).

See Open Table Formats on the Radar →

What to consider next

DII editorial view

• When selecting or renewing a platform, ask which open table formats it reads and writes natively
• Ask whether the catalogue is portable, not only the tables
• Test the features you rely on with each engine you use