Abstract
The Apache Iceberg REST Catalog specification, finalized in Iceberg 1.4 and gaining broad adoption through 2025, provides a vendor-neutral HTTP interface for catalog operations that decouples storage metadata from any single compute engine. As organizations run Spark, Trino, Flink, and DuckDB against the same table sets, the REST Catalog has emerged as the practical lingua franca for multi-engine lakehouse deployments. This article examines the protocol design, current implementation landscape, and operational considerations for teams migrating from Hive Metastore or proprietary catalog solutions.
Protocol Design and Scope
The REST Catalog specification defines a small surface area of HTTP endpoints covering namespace and table CRUD operations, credential vending, and commit conflict resolution. A catalog server exposes a base URI, and any compliant client - Spark’s iceberg-spark-runtime, Trino’s Iceberg connector, or the PyIceberg library - can authenticate and perform table operations without engine-specific configuration beyond the URI and credential. Credential vending is a notable addition: the catalog can return short-lived storage credentials scoped to a specific table and operation, removing the need for engines to hold long-lived cloud IAM keys.
The commit protocol relies on optimistic concurrency. Each write appends a new metadata file and then calls the catalog’s commitTable endpoint, supplying the expected base metadata location. If two engines commit simultaneously, the catalog rejects the second with a 409 Conflict, and the losing writer must refresh and retry. This model is simple but places the burden of retry logic on each engine’s connector, and behavior diverges slightly across implementations as of mid-2025.
Implementation Landscape
Several catalog servers implement the REST spec. Tabular (the commercial entity founded by the Iceberg creators) ships a managed REST Catalog. Apache Polaris, donated to the Apache Software Foundation in early 2025, is the leading open-source option; it adds principal roles and catalog-level access policies on top of the bare spec. Nessie continues to offer git-like branching semantics and now exposes a REST Catalog compatibility layer alongside its native API. AWS Glue Data Catalog added REST Catalog support in late 2024, allowing Athena, EMR, and third-party engines to share the same table registry.
PyIceberg 0.7 made the REST Catalog its default catalog type in local documentation examples, signaling the community’s intent to standardize tooling around it. DuckDB’s iceberg extension, still maturing through early 2025, supports read-only REST Catalog access and is widely used for ad-hoc analysis against production Iceberg tables without spinning up a JVM.
Operational Considerations
Teams adopting a REST Catalog face decisions that did not arise with Hive Metastore. Catalog high availability requires persistent state storage - Polaris uses a relational backend, and operators must plan for failover. Because the catalog participates in every write commit, it sits on the critical path for ingestion throughput; horizontal scaling and connection pooling deserve early capacity planning. Schema evolution and partition spec changes remain backward compatible within Iceberg’s design, but catalog-level authorization policies must be updated in lockstep when table structures change significantly.
Migration from Hive Metastore is non-trivial at scale. Tools such as the iceberg-hive-catalog bridge allow engines to continue reading through Hive while writing through REST during a transition window, but teams report that validating metadata parity across millions of partitions demands dedicated pipeline work before cutover.
Outlook
The REST Catalog specification is still evolving. Open proposals as of mid-2025 address multi-table transactions, view support aligned with the Iceberg Views spec, and richer audit event streaming. The trajectory is toward a catalog that handles not just table metadata but the full governance surface - access, lineage, and change notification - through a single HTTP interface usable by any engine that chooses to implement it.