10 Operationalizing Apache Iceberg

Orchestration turns Iceberg maintenance from manual, ad hoc work into repeatable, reliable, and observable workflows. Production Iceberg tables continuously accumulate snapshots, small files, delete files, manifests, and orphaned artifacts. Without automation, maintenance may run too often and waste resources, or too rarely and allow query latency, storage cost, and compliance risk to grow silently.

Tools such as Apache Airflow, dbt, Dagster, Argo Workflows, Prefect, or custom workflow engines can schedule and coordinate procedures like rewrite_data_files, expire_snapshots, rewrite_position_delete_files, and remove_orphan_files.

A DAG, or Directed Acyclic Graph, is a workflow structure where tasks run in a defined order without loops. In an Iceberg maintenance workflow, a DAG can ensure that one task completes before dependent tasks begin.

For example, an Airflow DAG might first query Iceberg metadata to check for small files, then run compaction only if a threshold is exceeded, then expire old snapshots, and finally remove orphan files. This makes maintenance predictable, auditable, and easier to debug.

Iceberg exposes metadata views such as files, snapshots, manifests, history, and refs. These can be queried like tables and used to decide when maintenance is actually needed.

For example, the files metadata table can reveal how many files are below a target size, such as 20 MB. If the count exceeds a threshold, orchestration can trigger compaction. The snapshots table can reveal snapshot sprawl, while delete file metrics can indicate when Merge-on-Read tables need delete file rewriting or compaction.

Different tables have different operational needs. A high-velocity CDC table may generate thousands of small files and frequent delete files, requiring frequent compaction and snapshot expiration. A monthly reporting table may only need lightweight maintenance after each monthly batch load.

Per-table policies allow teams to tune compaction frequency, snapshot retention, file size thresholds, orphan cleanup, and delete file rewriting based on table size, ingestion pattern, query importance, and compliance requirements. These policies can be stored in configuration files, catalog annotations, or table properties such as write.target-file-size-bytes, history.expire.min-snapshots-to-keep, and history.expire.max-snapshot-age-ms.

A production Iceberg lakehouse should monitor metadata, orchestration, storage, and query engine behavior. Useful Iceberg metadata metrics include small file counts, snapshot counts, manifest growth, delete file accumulation, partition skew, and orphan file trends.

Engine-level metrics from Spark, Trino, Dremio, or Flink should also be tracked, including query planning latency, scan time, task failures, and resource contention. Orchestration logs from Airflow, dbt, or Dagster should be monitored for failed DAGs, skipped jobs, retry loops, and abnormal task durations. These metrics can be exported to tools such as Prometheus, CloudWatch, or Datadog for dashboards and alerts.

Iceberg creates a new snapshot for every table mutation, including appends, overwrites, deletes, updates, and schema changes. These snapshots form a built-in audit trail that records what changed, when it changed, and which files were added or removed.

Auditors and engineers can query the snapshots, history, manifests, and metadata_log_entries metadata tables to investigate changes. Snapshot summaries can reveal record counts added or removed, operation types, commit times, and lineage. This makes it possible to answer questions such as “What job changed this table?”, “When did row counts drop?”, or “Which files were affected by an overwrite?”

Branches and tags provide version-control-like governance for Iceberg data. At the table level, Iceberg supports named snapshot references. A tag points to a fixed snapshot, making it useful for reproducible reporting or compliance checkpoints. A branch can move forward to newer snapshots, making it useful for development, QA, or controlled experimentation.

Catalog-level systems such as Project Nessie extend this idea across multiple tables, allowing teams to create isolated branches of an entire data environment. This supports testing, approval workflows, reproducibility, and safer promotion of changes into production. However, branches and tags must be governed carefully because referenced snapshots cannot be expired while they remain pinned.

Retention in Iceberg is mainly enforced through snapshot expiration and orphan file cleanup. The expire_snapshots procedure removes snapshots that are no longer needed and makes unreferenced data or metadata files eligible for deletion. The remove_orphan_files procedure removes files that are not referenced by any valid Iceberg metadata, often left behind by failed or aborted writes.

Snapshot expiration should generally run before orphan file cleanup. Retention policies should vary by workload: staging tables may expire snapshots aggressively, while regulated production tables may retain snapshots for weeks or months, or pin them with tags for audit purposes.

In Iceberg, a delete is first a logical change. In Copy-on-Write mode, affected data files are rewritten to exclude deleted rows. In Merge-on-Read mode, deletes are tracked separately using position delete files or deletion vectors, while the original data files may still physically contain the deleted rows.

For secure physical deletion, teams must coordinate the full lifecycle: issue the delete, compact or rewrite data files when necessary, expire snapshots that still reference old files, and clean up orphaned files. Branches, tags, retained snapshots, and delete files can all delay physical removal. This is especially important for GDPR, CCPA, and other compliance-driven deletion requirements.

Disaster recovery for Iceberg must protect the catalog, metadata files, and data files. The catalog maps table names to current metadata locations, so catalog loss can make a table inaccessible even if all files still exist. Recovery often depends on restoring catalog state or re-registering a table with a known metadata.json file using procedures such as register_table.

Best practices include enabling object storage versioning, replicating data and metadata across regions, backing up catalog mappings, validating snapshot integrity, testing rollback_to_snapshot, and running regular recovery drills. Recovery workflows should be automated through tools like Airflow or Dagster and aligned with RPO and RTO requirements.