6 Implementing the catalog layer

Apache Iceberg’s catalog layer is presented as the metadata backbone of a lakehouse: it registers and discovers tables, organizes namespaces, points engines to the current table metadata, and coordinates transactional commits. Because every query, schema change, write, rollback, and time-travel operation depends on accurate metadata resolution, the catalog is a strategic architectural component rather than a simple lookup service. It enables Iceberg’s separation of storage, compute, and metadata management, allowing engines such as Spark, Flink, Trino, Dremio, and others to share consistent table state while supporting ACID behavior through snapshot-based updates and optimistic concurrency control.

The chapter emphasizes that catalog selection should begin with organizational requirements gathered during platform audits. Key evaluation dimensions include performance, availability, scalability, governance, security, compliance, deployment flexibility, ecosystem compatibility, cost, operational overhead, and federation needs. Strong catalogs should support low-latency metadata access, high availability, fine-grained authorization, integration with enterprise identity systems, audit logging, lineage, encryption, credential vending, and automation-friendly APIs. For distributed enterprises, catalog federation and mesh-style architectures become important because they allow multiple catalogs to operate across regions, clouds, and business units while preserving local autonomy, regulatory compliance, and global discoverability.

The chapter also explains how the Apache Iceberg REST Catalog specification improves interoperability by replacing fragmented, language-specific catalog clients with a common HTTP-based interface for operations such as listing namespaces, loading table metadata, registering tables, and committing changes. It then surveys major catalog options and their trade-offs: simple built-in options like Hadoop, Hive, and JDBC; open source service-based catalogs such as Apache Polaris, Nessie, Gravitino, and Lakekeeper; and managed or platform-integrated offerings such as AWS Glue Data Catalog, Dremio Catalog, Snowflake Open Catalog, and Databricks Unity Catalog. The closing scenarios show that there is no universally best catalog: teams migrating from Hive may start with Hive Catalog, AWS-focused teams may prefer Glue, governance-heavy enterprises may choose Gravitino, branch-based testing workloads may benefit from Nessie, Dremio-centered migrations may use Dremio Catalog, and lightweight Python-first teams may begin with Hadoop Catalog before evolving toward REST-based or enterprise-grade solutions.

The catalog enables tools that access lakehouse tables to verify permissions and locate the corresponding data within the lake.

An Apache Iceberg catalog maps each table name to the location of its latest metadata file, which serves as the entry point for tools to begin scanning and interacting with the table.

Optimistic commits allow engines to write data and metadata and determine if the commit is successful by checking its place in the commit sequence before and after writing files and then committing.

Engines require tables from catalogs and return their metadata location.

The metadata provides different levels of data for an engine to identify a snapshot and determine which data files in that snapshot should be scanned.

Catalogs can have different levels of access controls, allowing you to control who has access to what namespaces or tables.

Using the REST Catalog spec, a client written in any language can talk to all catalogs regardless of what language the catalog server is written in, improving interoperability.

Before the REST catalog spec, each catalog service would have to maintain its own client libraries in each language, causing a hard-to-maintain and fragmented environment.

With the REST Catalog Spec, the same client library can be used for all catalogs, reducing the fragmentation and improving interoperability between iceberg catalogs.

With Hadoop catalog, a file called VERSION-HINT.TEXT is used to discover the metadata file instead of a catalog service, which you would send a request to.

• The catalog layer serves as the metadata backbone of an Apache Iceberg lakehouse, enabling reliable data discovery, governance, and interoperability.
• Built-in catalogs like Hadoop and JDBC offer simplicity but limited flexibility for scaling or integration.
• Open-source and self-managed catalogs such as Polaris, Gravitino, Nessie, and Lakekeeper provide customization and advanced features for complex deployments.
• Vendor-managed catalogs like AWS Glue Data Catalog, Dremio Catalog, and Snowflake Open Catalog deliver turnkey infrastructure with integrated security and enterprise support.
• The Apache Iceberg REST Catalog specification enables standardized communication between compute engines and catalogs, improving multi-engine compatibility.
• Catalog selection should align with your platform’s operational model, compliance requirements, cloud strategy, and future growth plans.
• Cloud-agnostic catalogs are increasingly important for organizations planning multi-cloud or hybrid deployments.
• Catalog technologies and integrations are evolving rapidly, so revisiting decisions as your architecture matures is essential.s

FAQ