Appendix A The metadata tables

Apache Iceberg metadata tables are special system tables that expose an Iceberg table’s internal metadata so it can be queried like regular data. They provide visibility into snapshots, commit history, data files, delete files, manifests, partitions, branches, tags, and historical table evolution.

They help teams monitor table health, diagnose performance problems, audit changes, support time travel investigations, and automate maintenance tasks such as compaction, snapshot expiration, and orphan file cleanup.

In Spark, append the metadata table name to the base table name. For example:

SELECT * FROM db.table.history;
SELECT * FROM db.table.snapshots;
SELECT * FROM db.table.files;
SELECT * FROM db.table.partitions;

In Dremio, metadata tables are commonly accessed through table functions. For example:

SELECT * FROM TABLE(table_history('my_table'));
SELECT * FROM TABLE(table_snapshot('my_table'));
SELECT * FROM TABLE(table_files('db.table'));
SELECT * FROM TABLE(table_partitions('db.table'));

The history table shows the commit timeline and lineage of snapshots, including when a snapshot became current, its parent snapshot, and whether it is part of the current table history. The snapshots table provides operational details for each snapshot, such as the operation type, commit time, manifest list, and summary metadata.

Use the files metadata table. It exposes file-level information such as file_path, record_count, file_size_in_bytes, file format, partition details, and column-level metrics. A common query is:

SELECT file_path, file_size_in_bytes
FROM db.table.files
WHERE file_size_in_bytes < 134217728;

This identifies files smaller than 128 MB that may be candidates for compaction.

Use the partitions metadata table. It provides metrics such as record_count, file_count, total_data_file_size_in_bytes, delete file counts, and the last updated snapshot for each partition. For example:

SELECT partition, record_count, total_data_file_size_in_bytes
FROM db.table.partitions
ORDER BY record_count DESC
LIMIT 10;

Large differences across partitions may indicate skew and suggest a need for compaction, clustering, or repartitioning.

Manifest files are intermediate metadata files that list groups of data and delete files used by a snapshot. The manifests table helps detect metadata sprawl, small manifests, inefficient write patterns, and partition distribution issues. A growing number of manifests can slow query planning and may indicate the need for metadata compaction.

The refs metadata table shows named references associated with an Iceberg table, including branches and tags. Tags are immutable references to specific snapshots, often used for audit or compliance. Branches are mutable references that can evolve independently, supporting workflows such as experimentation, write-audit-publish, and data recovery. The table also exposes retention settings that affect snapshot expiration.

Metadata tables can drive threshold-based maintenance instead of relying only on fixed schedules. For example, use files to trigger compaction when too many files are below a target size, snapshots to trigger expiration when snapshot counts or ages exceed limits, manifests to detect metadata bloat, and refs to ensure important branches and tags are protected during cleanup.