9 Maintaining an Iceberg lakehouse
Maintaining an Apache Iceberg lakehouse requires ongoing operational discipline after the initial architecture and deployment are complete. As tables grow and workloads evolve, performance can degrade because of small files, poorly colocated data, fragmented metadata, and accumulated delete files in Merge-on-Read workloads. These issues increase file I/O, slow query planning, reduce data skipping effectiveness, and raise storage and compute costs. Iceberg’s metadata tables are central to diagnosing these problems because they expose file sizes, manifests, snapshots, delete files, partition layout, and table history through SQL-accessible views.
The chapter presents compaction as the main technique for restoring efficient table layout. Compaction rewrites data and metadata to merge small files, improve clustering or sorting, apply delete files to base files, consolidate manifests, and refresh file statistics. It can be performed with Iceberg procedures such as rewriting data files and manifests, or with engine-specific commands such as Dremio’s optimization features. Effective compaction depends on choosing appropriate target file sizes, selecting only the files that need rewriting, and using filters to scope jobs by partition, time range, or workload priority. This allows teams to optimize hot or degraded parts of a table incrementally without disrupting service-level agreements or consuming excessive compute resources.
The chapter also explains how storage footprint, retention, compliance, and access controls fit into lakehouse maintenance. Iceberg’s snapshot isolation and time travel retain historical data and metadata until snapshots are expired and unreferenced files are cleaned up, so teams must align snapshot expiration with recovery needs, cost control, and regulations such as GDPR or CCPA. Copy-on-Write and Merge-on-Read tables require different retention strategies, especially because MOR delete files can prolong storage usage and hurt read performance unless compacted regularly. Finally, the chapter emphasizes layered security across storage, catalog, and engine levels: object-store permissions provide a baseline, catalog-level roles and credential vending centralize governance, and fine-grained row or column controls currently depend on engine capabilities while the ecosystem moves toward more interoperable policy enforcement.
More file IO operations are needed when the same data is stretched across a bigger number of files, resulting in slower performance. The fewer IO operations, the faster the query can complete.
If you were to search all “yellow” records, you’d have to read all three files, which contain other unnecessary data for the query.
with better colocation of data, a query for all “yellow” records only needs to read one file instead of all three files. Partitioning allows records with unique values to be written to distinct data files.
Metadata bloat occurs when you have more manifests than necessary relative to the number of data files, or you have manifests tracking too many small data files. Both of these will result in delays of query planning.
Ideally, you have fewer manifest files tracking fewer data files containing a greater number of records per file, minimizing the cost of metadata traversal and data file IO.
In Copy-on-Write, when you delete records, the data file with those records is rewritten without them. In Merge-on-Write, instead of rewriting a whole file, the records deleted are tracked in a delete file or deletion vector, which can be used to reconcile when the table is read, which enables faster writing to the table but slows down reading.
During compaction, data files will be written so there are fewer larger files, and manifests will be rewritten so fewer manifests are tracking more files. This will improve the performance of tables that have seen degraded performance after many writes.
Iceberg tables have snapshot isolation, allowing you to query older versions of the table. The catch is that all the historical data is still being stored along with the current data.
When Snapshot 1 is expired, only metadata or data files that aren’t associated with the remaining snapshots are deleted.
Summary
- Iceberg manages physical files through structured, linked metadata files, making maintenance a critical operation.
- Compaction is essential for managing metadata overhead, merging changes, removing deleted rows and improving read performance, especially with Merge-on-Read tables.
- Snapshot expiration and garbage collection prevent unbounded metadata and file storage growth.
- Effective file sizing and clustering strategies help optimize query performance and reduce scan costs.
- Cloud-native IAM policies should be scoped to temporary credentials for read/write access and tightly controlled with IAM conditions.
- Credential vending enables catalogs to serve as the authoritative access control layer for both metadata and data storage.
- Polaris-style RBAC models separate users (principals) from permissions (catalog roles), providing scalable access control patterns.
- Engines currently lack a standardized mechanism for enforcing fine-grained access control (FGAC).
- A proposed FGAC model for Iceberg introduces catalog-issued row filters and column transformations that engines can enforce at runtime.
- Future interoperability for FGAC across engines depends on standardized protection expression formats like SQL or Substrait.
FAQ
Why does an Apache Iceberg lakehouse need ongoing maintenance?
An Iceberg lakehouse needs maintenance because table performance can degrade as data grows, write patterns change, and metadata accumulates. Common problems include small files, poorly colocated data, metadata sprawl, and accumulated delete files in Merge-on-Read tables. Without proactive maintenance, query planning slows down, storage costs increase, and governance or compliance requirements become harder to meet.
What are small files, and why do they hurt Iceberg query performance?
Small files are data files that are much smaller than the desired target size, often created by frequent streaming or micro-batch ingestion. Query engines such as Spark, Trino, and Dremio must open, scan, and coordinate work for each file. When a table contains thousands of small files, file I/O, cloud storage API calls, metadata reads, and query planning overhead can dominate query time. Small files also increase Iceberg metadata size because each file must be tracked in manifests.
How can I detect small file problems in an Iceberg table?
You can detect small file problems by querying Iceberg metadata tables, especially the files metadata table. This table exposes file-level details such as file size, partition, path, and statistics. A high number of files below a practical threshold, such as 128 MB or your chosen target size, may indicate that the table would benefit from compaction.
What is poorly colocated data, and how does it affect query performance?
Poorly colocated data means records that are commonly queried together are scattered across many files or partitions. For example, if a sales table is partitioned by region but frequently queried by customer_id, the engine may be unable to prune irrelevant files efficiently. This increases file scans, CPU usage, and latency. Better partitioning, sorting, clustering, and targeted compaction can improve locality so queries read fewer files and less irrelevant data.
What is metadata sprawl in Apache Iceberg?
Metadata sprawl occurs when Iceberg’s metadata layer becomes fragmented or excessively large, especially through too many manifest files or manifests that track many small data files. Iceberg snapshots point to manifest lists, which point to manifest files, which track data files. Frequent small commits, streaming ingestion, or high-frequency writes can create many small manifests. This increases query planning time and adds overhead to operations such as snapshot expiration, rollback, and schema evolution.
How do you fix metadata sprawl in Iceberg?
Metadata sprawl is addressed by rewriting and consolidating manifests. In Spark, you can use the Iceberg procedure rewrite_manifests. In tools such as Dremio, metadata optimization may be included through commands such as OPTIMIZE with manifest rewriting support. Regular metadata compaction helps reduce unnecessary manifest traversal, improves query planning, and keeps table operations responsive.
What is compaction in Apache Iceberg?
Compaction is the process of rewriting Iceberg data and metadata files into a more efficient layout. It can merge small files into larger files, split oversized files, apply delete files to base files in Merge-on-Read tables, sort records for better locality, update file statistics, and consolidate manifests. Compaction improves query performance by reducing file counts, improving data skipping, lowering metadata overhead, and restoring read efficiency after many writes.
What is the difference between bin-pack and sort compaction?
Bin-pack compaction is the default strategy. It combines small files into files closer to a target size, reducing file-open overhead and metadata overhead. Sort compaction goes further by rewriting records in a defined order, such as by event_date, customer_id, or using Z-order across multiple columns. Sort compaction can improve data locality and make predicate pruning more effective for filtered queries.
How should I choose a target file size for Iceberg compaction?
Iceberg’s default target file size is often 512 MB, configured through write.target-file-size-bytes, but the best size depends on the workload and query engine. Some environments benefit from smaller files, while engines such as Dremio or Trino may handle larger files efficiently. Consider query engine behavior, compression, table width, nested data, row group size, available memory, and compaction cost. The target is a guide rather than a strict guarantee, so teams should monitor query plans, I/O throughput, spill behavior, and parallelism to tune file sizes over time.
What is snapshot expiration, and why is it important?
Snapshot expiration removes old Iceberg snapshots and deletes data or metadata files that are referenced only by expired snapshots. This reduces storage footprint and metadata overhead while preserving files still needed by active or retained snapshots. In Spark, snapshot expiration can be run with expire_snapshots. In Dremio, it can be run with VACUUM TABLE ... EXPIRE SNAPSHOTS. Expiration policies should align with recovery windows, audit needs, compliance requirements, and the desired time travel period.
Architecting an Apache Iceberg Lakehouse ebook for free
