Appendix C The Apache Iceberg specification

The Apache Iceberg specification is a versioned, living standard that defines how Iceberg tables behave. It describes how table metadata is structured, how snapshots are created and updated, how manifests track files, how deletes are represented, how schemas and partitions evolve, and how concurrency and commits are managed. Its purpose is to ensure that different engines, such as Spark, Flink, Trino, and Dremio, can reliably read and write the same Iceberg tables.

Iceberg formalizes table behavior to avoid ambiguity, inconsistent engine behavior, data corruption, and vendor lock-in. By defining a shared contract, Iceberg provides predictable reads and writes, isolation between readers and writers, backward-compatible schema and partition evolution, and consistent snapshot tracking. This makes Iceberg suitable for multi-engine, cloud-native, and multi-tenant lakehouse environments.

Version 1 introduced the core Iceberg model: immutable file tracking, snapshots, manifest files, schema evolution, and partition independence. Version 2 added row-level deletes through position deletes and equality deletes, along with stricter writer validation rules. Version 3 expanded the format with extended data types, default column values, multi-argument transforms, row lineage tracking, deletion vectors, and table encryption metadata.

Iceberg version 1 introduced the foundation for metadata-driven analytical tables. Instead of relying on directory layouts or file naming conventions, v1 tracks data files explicitly through manifests. It supports snapshot-based access, safe schema evolution, partition values stored in metadata, and data files in Parquet, Avro, and ORC formats. These features enabled consistent reads, efficient query planning, and safer table evolution.

Iceberg version 2 supports row-level deletes without requiring full data file rewrites. It does this through delete files. Position deletes identify rows by file path and row position, while equality deletes identify rows by matching column values, such as user_id = 42. During query planning and execution, engines apply these delete files so that deleted rows are excluded from results.

Deletion vectors are a version 3 feature that efficiently tracks rows that have been logically deleted. Instead of relying only on separate delete files during reads, deletion vectors use compact binary structures to mark deleted rows. This can improve read performance and supports more advanced data management use cases, especially where row-level changes are frequent.

Each Iceberg table has immutable metadata files that describe the full table state, including schema, partition spec, properties, and the current snapshot. When a change is committed, Iceberg writes a new metadata file and updates the catalog pointer atomically. Readers either see the old table state or the new one, never a partial update. Snapshots provide complete, immutable views of the table at points in time, enabling time travel, rollback, auditing, and snapshot isolation.

A snapshot points to one or more manifest lists. A manifest list acts as an index of manifest files and records metadata such as manifest location, partition range, snapshot ID, and sequence number. Each manifest file lists a subset of the table’s data files, along with partition statistics and file-level metrics. This layered metadata structure helps engines prune unnecessary files and plan scans efficiently, even for tables with millions of files.

The REST Catalog specification defines a standard HTTP-based API for interacting with Iceberg catalogs. It allows engines to manage namespaces, create and load tables, update metadata, commit transactions, gather metrics, and configure catalog behavior through consistent JSON-based endpoints. Its goal is to let any Iceberg client communicate with any compliant REST Catalog server without custom connectors or vendor-specific logic.

Puffin is a compact, binary, versioned file format for storing auxiliary Iceberg metadata outside the core table metadata structures. It is commonly used for column-level metrics, bloom filters, custom indexes, histograms, quantiles, or cardinality estimates. Puffin files are optional, so tables work normally without them, but when present they can improve scan planning, observability, and advanced analytics without bloating manifests or snapshot metadata.