ACE Journal

Lakehouse Governance with Open Table Formats

Who this is for: data platform engineers and data architects who already run a lakehouse on object storage, Parquet, and at least one query engine, and who have just been handed a governance requirement. Maybe a data contract that downstream teams keep breaking, maybe an auditor asking who changed a table last quarter, maybe a regulation that says one team must not see another team’s rows. The assumption is that you know what a Parquet file is and have written a Spark or Trino query, but you have not yet treated the table format as the place governance lives. This is the article that maps each requirement to the layer that actually answers it.

The Hive Metastore could not tell you who broke the table

Picture the failure that put open table formats on the roadmap of every serious data platform. A dashboard breaks at nine in the morning, the numbers wrong since some point overnight. The on-call data engineer finds the table and asks the only question that matters: what changed, and who changed it? In a classic Hive-on-S3 lakehouse the answer is silence. The table is a directory of Parquet files registered in a Hive Metastore that records schema and partition layout but keeps no history. No commit log, no list of writers, no way to see the table as it was the night before. The engineer reconstructs the timeline from object-store access logs, Airflow runs, and guesswork, and the post-mortem takes a day.

That gap is not an operational annoyance, it is a governance hole. You cannot enforce a contract you cannot observe, audit a change you did not record, or prove to a reviewer that an unauthorized write never happened when the table itself remembers nothing. The Hive format treated the directory of files as the source of truth and the metastore as a thin index over it, which works fine until you need the table to account for itself.

Open table formats close the hole by inverting the relationship. Instead of files plus a thin index, the table becomes a metadata tree that points at the files, and that tree records every commit: when it happened, which principal made it, which files it touched, and what the schema was at that instant. Apache Iceberg, Delta Lake, and Apache Hudi each implement this differently, but all three turn the table into something that can answer “what changed and who changed it” without a forensic exercise. Governance becomes a property the table carries, not a system you build alongside it.

The rest of this article walks the layers: the mechanics of the metadata tree, a worked example of the three governance primitives the format gives you directly (schema enforcement, snapshot audit, time travel), the hard part that the format does almost nothing about access control and why that is correct, how a REST catalog and query engine complete the picture, where the three formats differ, and the half-dozen ways the arrangement breaks in production.

Mechanics: the table is a tree of immutable metadata

The core trick shared by all three formats is that a table is no longer a mutable directory. It is a stack of immutable metadata files culminating in a single pointer, and a commit is an atomic swap of that pointer. Nothing in the tree is ever edited in place: a write produces new data files and new metadata, then atomically advances the table’s current state to point at it. Readers always see a complete, consistent snapshot, never a half-written table.

In Iceberg the tree has three layers above the data. At the top, a metadata file records the current schema, partition spec, and the full list of snapshots. Each snapshot points at a manifest list, which points at manifest files, which point at the Parquet data files plus per-file statistics. Figure 1 shows the structure and the one pointer that makes a commit atomic.

flowchart TD
    Cat["catalog pointer<br/>(current metadata)"] --> Meta["metadata.json<br/>schema + snapshot list"]
    Meta --> S2["snapshot N<br/>(current)"]
    Meta --> S1["snapshot N-1<br/>(retained for history)"]
    S2 --> ML["manifest list"]
    ML --> M1["manifest file"]
    M1 --> D1["data.parquet"]
    M1 --> D2["data.parquet"]
    S1 -. older files .-> M1

Figure 1. The Iceberg metadata tree. Each table state is an immutable snapshot, and a commit atomically swaps the single catalog pointer from snapshot N-1 to N. What to look for: the catalog holds exactly one pointer, which is why a commit is all-or-nothing and why the previous snapshot stays readable for history and time travel.

Two properties fall out of this design and both are governance primitives. First, because the pointer swap is atomic and the catalog arbitrates it, concurrent writers cannot corrupt the table: the loser of a race retries against the new snapshot. That gives you serializable or snapshot isolation depending on the engine, which means a contract check at commit time is meaningful rather than racy. Second, because old snapshots are retained until you expire them, the table holds its own history. The audit trail is not a feature someone added, it is a side effect of never mutating in place.

Delta Lake reaches the same place differently. Instead of a metadata tree it keeps an ordered transaction log, JSON files numbered in commit order, periodically compacted into Parquet checkpoints. Reading a Delta table means replaying the log, and like Iceberg’s snapshots it records operation, predicate, and writer metadata per commit. Hudi, designed first for streaming upserts, organizes around a timeline of commit instants with copy-on-write and merge-on-read storage so high-frequency updates do not rewrite whole files. Different data structures, same governance consequence: the table records its own change history.

Worked example: the three primitives the format gives you for free

Here are the governance capabilities you get from the format itself, catalog and engine left out for now. The examples use Iceberg through Spark SQL, with the Delta equivalent noted.

Schema enforcement and evolution as a contract

The first primitive is a schema contract enforced at commit time. Iceberg distinguishes safe evolution from breaking change. Adding a nullable column, widening an int to a long, or reordering columns are compatible and tracked by stable per-column field IDs, so a rename is a metadata operation rather than a destructive rewrite. A write that violates the declared schema (wrong type, a null in a required column) is rejected when it tries to commit, not discovered three dashboards downstream.

-- Declare the contract once.
CREATE TABLE lake.sales.orders (
    order_id    BIGINT  NOT NULL,
    customer_id BIGINT  NOT NULL,
    amount      DECIMAL(12,2),
    placed_at   TIMESTAMP)
USING iceberg
PARTITIONED BY (days(placed_at));

-- Safe evolution: a new nullable column, tracked by field ID.
ALTER TABLE lake.sales.orders ADD COLUMN currency STRING;

-- This commit is rejected: amount is not nullable-typed as text.
-- INSERT INTO lake.sales.orders
--   VALUES (1, 42, 'twenty', current_timestamp(), 'USD');

The contract lives in the table, so every engine that writes through it inherits the same enforcement. In Delta the guarantee is on by default through schema-on-write, exposed as a table property you can relax for exploratory loads. The governance value is that “the schema is the contract” stops being a wiki page and becomes a runtime invariant.

Snapshot history as an always-on audit trail

The second primitive answers the nine-a.m. dashboard question. Every commit is recorded with a timestamp, the writing principal, the operation, and a summary of changed files, and that record is queryable as a table.

-- Iceberg: full snapshot chain, newest last.
SELECT made_current_at, snapshot_id, operation, summary
FROM   lake.sales.orders.snapshots
ORDER  BY made_current_at;

-- Iceberg: who wrote, what files moved, per commit.
SELECT committed_at, operation, summary['spark.app.id'] AS app
FROM   lake.sales.orders.history;

The Delta equivalent is DESCRIBE HISTORY lake.sales.orders, returning operation, user, and predicate for each version. This log is not a substitute for a dedicated security audit system, and the hard-part section explains why, but it is a lightweight, always-on trail that answers a large fraction of internal compliance questions with no extra infrastructure. The question that took a day of forensics on Hive is now one query.

Time travel for reproducibility and rollback

The third primitive follows from retained snapshots: you can read the table as of any point still in history, by timestamp or snapshot ID. That turns “the numbers were wrong overnight” into a computable diff and a bad batch load into a one-statement rollback.

-- Read the table as it was before the bad overnight load.
SELECT * FROM lake.sales.orders
TIMESTAMP AS OF '2026-05-09 23:00:00';

-- Pin a training run to an exact snapshot for reproducibility.
SELECT * FROM lake.sales.orders VERSION AS OF 3921719215861073152;

-- Roll the live table back to a known-good snapshot.
CALL lake.system.rollback_to_snapshot('sales.orders', 3921719215861073152);

Reproducibility is a governance requirement in its own right. A model trained on “the orders table” is unauditable, one trained on “the orders table at snapshot 3921719215861073152” can be re-derived and defended. Schema enforcement, snapshot audit, and time travel are the three things the format genuinely gives you. The next section is about what it deliberately does not.

The hard part: the format does not enforce who can read a row, and that is correct

The non-obvious lesson, the one practitioners learn by getting it wrong, is that the table format is not an access-control system, and trying to make it one is a category error. The format records what the data is and how it changed. It does not know who is asking or what they are allowed to see. Row-level security and column masking are policy, policy needs an identity to evaluate against, and the table format has no concept of identity. That belongs to the catalog and the query engine.

Figure 2 is the separation worth memorizing: three layers, three responsibilities, and almost every governance mistake in a lakehouse is a layer doing a job that belongs to a different one.

flowchart TB
    subgraph Engine["query engine (Trino / Spark / Flink)"]
        E["evaluates row filters + column masks<br/>at scan-plan time, per identity"]
    end
    subgraph Catalog["REST catalog (Polaris / Unity / Nessie)"]
        C["owns identity, grants, policies<br/>arbitrates atomic commits"]
    end
    subgraph Format["table format (Iceberg / Delta / Hudi)"]
        F["stores schema, data files,<br/>immutable snapshot history"]
    end
    E -->|asks for policy| C
    C -->|hands back table + rules| E
    C -->|reads / advances pointer| F

Figure 2. The three-layer division of responsibility in a governed lakehouse. The format stores and remembers, the catalog decides who may do what, the engine enforces it during scan planning. What to look for: access policy lives in the catalog and is applied by the engine, never in the file format, so masking a column means configuring the catalog, not editing the table.

The payoff of putting policy in the catalog is that it can be evaluated at scan-planning time. When a query arrives, the engine asks the catalog for the policies that apply to this principal, and a row-level filter becomes a predicate that prunes whole data files before any bytes are read. With Iceberg this composes cleanly with manifest-level statistics: a filter for region = 'EU' skips every file whose min and max say it holds no EU rows. The access policy and the physical layout cooperate, so security pushdown also makes the query faster and reduces how much restricted data leaves storage.

Here is what a policy looks like, expressed against the catalog rather than the table. Exact syntax varies by catalog, this is the Unity-Catalog-flavored shape that has become the de facto vocabulary:

-- A column mask: non-finance roles see only the last four digits.
CREATE FUNCTION mask_account(acct STRING)
RETURN CASE WHEN is_account_group_member('finance')
            THEN acct ELSE concat('****', right(acct, 4)) END;

ALTER TABLE lake.sales.orders
  ALTER COLUMN account_number SET MASK mask_account;

-- A row filter: regional analysts see only their own region's rows.
CREATE FUNCTION region_filter(region STRING)
RETURN is_account_group_member('global') OR region = current_region();

ALTER TABLE lake.sales.orders
  SET ROW FILTER region_filter ON (region);

Notice that none of this touched the data files or the Iceberg metadata. The orders table on disk is identical for the finance analyst and the regional analyst, the difference is entirely in what the catalog tells the engine to do at query time. That is the design working as intended. Try to enforce this instead with one physical table per region or pre-masked columns in separate files, and you have moved policy into the format, multiplied storage and maintenance, and guaranteed the copies will drift. Keep the format dumb about identity and let the catalog be smart about it.

The snapshot log earns the same caution. It is an excellent operational audit trail and a poor security audit log: by default any principal who can read the table can read its full history, and a real compliance system needs tamper-evidence and retention guarantees the format does not provide. Treat it as the first thing you reach for in an incident, not the thing you show an external auditor.

Integration: how the catalog ties it into a day-to-day platform

In practice the catalog turns three formats and four engines into one governed platform, and the REST catalog specification is what made that practical. Before it, every engine spoke to a Hive Metastore through engine-specific glue, and “the same table” in Spark was a separate registration in Trino, with permissions configured twice and drifting apart. The REST catalog, driven by the Iceberg community and implemented by Apache Polaris, Project Nessie, Unity Catalog’s open endpoint, and several cloud-managed offerings, defines one HTTP API for metadata operations. Register a table once, and Spark, Flink, Trino, and DuckDB all reach it through the same endpoint with the same policy applied. Figure 3 shows the day-to-day shape.

sequenceDiagram
    participant U as analyst
    participant E as Trino
    participant C as REST catalog (Polaris)
    participant S as object storage
    U->>E: SELECT ... FROM orders WHERE region='EU'
    E->>C: loadTable(orders) + policies for principal
    C-->>E: metadata pointer + row filter + column mask
    E->>E: plan scan, prune files, apply mask
    E->>S: read only matching EU data files
    S-->>E: parquet bytes
    E-->>U: filtered, masked rows

Figure 3. A governed read through a REST catalog. The engine resolves the table and its policies in one catalog round-trip, then prunes and masks before touching storage. What to look for: the catalog hands back both the metadata pointer and the access rules in the same call, so policy and physical planning happen together rather than as separate bolt-ons.

The workflow this enables is the one that matters. A data engineer registers a table and declares its schema contract once. The platform team grants access and attaches row filters and masks in the catalog, by group rather than individual, so policy survives people joining and leaving teams. Analysts query through whichever engine fits the job and all see the same enforced view. When something breaks, the on-call engineer reads snapshot history to find the offending commit and time-travels to confirm the pre-incident state. None of these requires a separate governance product, because the format and catalog cover the common cases between them. You add a dedicated governance tool when you outgrow the basics, not on day one.

For governance specifically, the three formats are more alike than the marketing suggests. All three give you atomic commits, schema evolution, snapshot or timeline history, and time travel. The differences that matter for a governance decision are about ecosystem and engine reach, not whether the primitives exist.

Apache Iceberg has the broadest multi-engine and multi-vendor support and drives the REST catalog specification the others are converging toward. Its hidden partitioning and field-ID schema evolution are the cleanest, and the open catalog ecosystem (Polaris, Nessie, cloud-managed REST catalogs) is most mature here. If your environment is genuinely multi-engine and catalog portability is a first-class property, Iceberg is the safe center of gravity.

Delta Lake has the most polished single-vendor story. Schema enforcement on by default, DESCRIBE HISTORY and RESTORE as first-class SQL, and the tightest Spark integration make it excellent inside a Databricks-centric platform. Unity Catalog gives a strong governance surface but has historically been more coupled to Databricks-managed compute than the open REST catalogs, so weigh portability if you run engines outside that orbit.

Apache Hudi comes from the streaming-ingest world and leads on high-frequency upserts and incremental processing through merge-on-read storage and record-level indexing. For a governance use case dominated by mutable, late-arriving data, a CDC stream that constantly updates customer records, Hudi does less file rewriting. Its multi-engine catalog story is less broad than Iceberg’s, so the trade is upsert efficiency against ecosystem reach.

The choice is rarely about a governance capability one format has and another lacks. It is about which engines you must support, whether catalog portability is a hard requirement, and whether your writes are batch-append or high-frequency upsert. Pick for your engines and write pattern, then do the governance work in the catalog, which is portable across all three to a meaningful degree.

Where it breaks

The arrangement fails in recognizable ways, most of them a layer asked to do a job that belongs to another.

Mistaking snapshot history for an audit system. Snapshot history records writes, not reads, so it tells you who changed sensitive data, not who looked at it. It is readable by anyone who can read the table and can be truncated by routine expiration. Present it to an auditor as a compliance log and the first probing question collapses it. Use it for operations, stand up a real audit system for compliance.

Snapshot expiration silently deleting your history. Snapshots and their files are retained only until you expire them, and maintenance jobs that compact and expire are not optional at scale or metadata grows without bound. The failure mode is an aggressive 7-day expiration window, then discovering you cannot time-travel to an incident that started 10 days ago. Retention is a governance policy, set it deliberately rather than accepting a maintenance-tuned default.

Policy enforced in the wrong layer. Under deadline pressure teams reach for the table when they should reach for the catalog: one physical table per tenant for “isolation,” pre-masked columns, regional copies for residency. Each works briefly then drifts, because the same logical data now lives in multiple physical truths kept in sync by hand. Row filters and masks belong in the catalog, evaluated at query time against one physical table.

Catalog lock-in undoing format openness. The format can be perfectly open while the catalog quietly becomes the lock-in point. Policies, grants, and lineage authored in a proprietary catalog do not travel when you switch engines or vendors, even though the Parquet and Iceberg metadata would. Evaluate the catalog’s portability (REST-spec compliance, policy export) as seriously as the format’s, since that is where lock-in accrues.

Concurrent-writer contention on hot tables. Optimistic concurrency is wonderful until many writers target the same partitions. Commits collide, losers retry against the new snapshot, and a hot table can spend more time retrying than writing. Nothing corrupts, but throughput craters. The fix is partitioning and write design that spreads commits, though it surfaces as “governance made our pipeline slow.”

Small-file and metadata bloat from streaming writes. High-frequency ingestion produces many small files and long manifest or log chains. Query planning slows as the engine reads more metadata, and the snapshot history that powers your audit trail becomes expensive to scan. Compaction and metadata maintenance are the cost of keeping the governance primitives fast, budget for them as ongoing platform work.

Further reading

Start with the Apache Iceberg specification [1] and its evolution documentation [2] for the metadata tree, field-ID schema evolution, and snapshot mechanics. The Delta Lake transaction-log protocol [3] and time-travel documentation [4] give the equivalent for Delta, and the Apache Hudi concepts overview [5] covers the timeline and copy-on-write versus merge-on-read. For the catalog layer, the Iceberg REST catalog OpenAPI specification [6] is the canonical contract, and Apache Polaris [7] and Project Nessie [8] are the open implementations. For access control as policy in the engine and catalog, the Unity Catalog row-filter and column-mask documentation [9] and the Trino Iceberg connector documentation [10] show how filters and masks apply at scan time. The original lakehouse paper [11] is the conceptual anchor for why this architecture exists at all.

Written with AI assistance, editorially reviewed.

References

[1] Apache Iceberg, “Iceberg Table Spec.” https://iceberg.apache.org/spec/

[2] Apache Iceberg, “Schema Evolution.” https://iceberg.apache.org/docs/latest/evolution/

[3] Delta Lake, “Delta Transaction Log Protocol.” https://github.com/delta-io/delta/blob/master/PROTOCOL.md

[4] Delta Lake, “Work with Delta Lake Table History.” https://docs.delta.io/latest/delta-utility.html

[5] Apache Hudi, “Hudi Concepts.” https://hudi.apache.org/docs/concepts/

[6] Apache Iceberg, “Iceberg REST Catalog API (OpenAPI).” https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml

[7] Apache Polaris, “Apache Polaris (Incubating).” https://polaris.apache.org/

[8] Project Nessie, “Nessie Documentation.” https://projectnessie.org/

[9] Databricks, “Filter Sensitive Table Data Using Row Filters and Column Masks.” https://docs.databricks.com/en/tables/row-and-column-filters.html

[10] Trino, “Iceberg Connector.” https://trino.io/docs/current/connector/iceberg.html

[11] M. Armbrust et al., “Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics,” CIDR 2021. https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf