Appendix B Python for Apache Iceberg
Apache Iceberg is presented as a core foundation for modern lakehouse architectures, and Python is shown as a flexible ecosystem for building, managing, and analyzing Iceberg data without relying on JVM-based tools such as Spark. The appendix explains how Python can act as both the control plane and execution layer for Iceberg workflows, covering schema definition, table creation, appends, overwrites, scans, queries, and integration with catalogs and object storage.
Several Python-friendly tools are explored for different Iceberg use cases. PyIceberg provides direct table and catalog management, including schema evolution, transactional writes, filtered scans, and metadata operations. Polars and DuckDB focus on fast analytics: Polars offers lazy, columnar dataframe operations with strong read performance and experimental writes, while DuckDB enables SQL-based querying, catalog attachment, inserts, transformations, and time-travel queries. Daft extends the ecosystem with distributed dataframe processing, efficient Iceberg reads through PyIceberg, Arrow interoperability, and workflows where processed data can be exported and registered back into Iceberg.
The appendix also covers higher-level platforms that expand Iceberg’s role in production lakehouses. Dremio provides scalable SQL access through ODBC and Arrow Flight, supporting both analytics and table modification operations such as create, insert, update, delete, and merge. Bauplan introduces Git-style branching, validation, schema evolution, and merge workflows for Iceberg-backed data, treating data engineering changes like version-controlled code. SpiceAI adds a federated query layer for analyzing Iceberg alongside other systems, making it useful for AI-ready analytics environments where governed Iceberg data must be queried together with external sources.
B.8 Summary and best practices
Throughout this appendix, you’ve explored how Python interacts with Apache Iceberg through a range of libraries, each with different strengths, abstractions, and levels of integration. While PyIceberg provides the direct and foundational interface for Iceberg table operations, higher-level frameworks like Polars, DuckDB, Daft, Dremio, Bauplan, and SpiceAI build on top of these principles to offer richer analytical and orchestration capabilities. Understanding how to combine these tools effectively is the key to maintaining a clean, scalable, and efficient Iceberg-based data platform.
At the heart of every Iceberg workflow in Python lies the same set of operations: creating, reading, writing, evolving schemas, and scanning data efficiently. Let’s summarize the best practices and patterns that emerge across all these integrations.
This example encapsulates the lifecycle that all Python-based Iceberg tools follow, whether the operations are executed through PyIceberg directly or abstracted through other engines.
Across all libraries discussed, several best practices consistently lead to more reliable and efficient Iceberg workflows:
- Use catalogs consistently.
- Leverage schema evolution wisely.
- Optimize writes upfront.
- Prefer lazy and pushdown-enabled scans.
- Adopt branch-based experimentation.
- Unify analytics and AI access.
Apache Iceberg’s design allows these diverse tools to interoperate seamlessly. Whether you’re managing data ingestion, optimizing queries, or powering AI applications, the combination of Python and Iceberg provides both flexibility and control. Together, they form a powerful foundation for an open, versioned, and fully governed lakehouse ecosystem.
FAQ
What is the main goal of using Python with Apache Iceberg in this appendix?
The goal is to show how to build, manage, and analyze Apache Iceberg lakehouse data entirely in Python, without relying on JVM-based systems such as Spark. The appendix demonstrates how Python tools can define schemas, create tables, append and overwrite data, scan tables, and run analytical queries.Which Python libraries and frameworks are covered for working with Apache Iceberg?
The appendix covers PyIceberg, Polars, DuckDB, Daft, Dremio, Bauplan, and SpiceAI. Each tool plays a different role: PyIceberg provides direct Iceberg table and catalog access, Polars and DuckDB support fast analytics, Daft adds distributed dataframe processing, Dremio provides scalable SQL execution, Bauplan adds Git-style version control, and SpiceAI enables federated analytics.What is PyIceberg used for?
PyIceberg is the official Python client for Apache Iceberg. It allows Python users to connect to Iceberg catalogs, create namespaces and tables, append data, overwrite partitions, evolve schemas, and scan Iceberg tables directly from Python without requiring Spark or a JVM runtime.How can PyIceberg create and load data into an Iceberg table?
PyIceberg can connect to a catalog, create a namespace, create a table using a schema, and append data from a PyArrow table. For example, a local SQL catalog backed by SQLite can be loaded withload_catalog, a table can be created from a Parquet schema, and data can be appended using table.append(arrow_table).How does PyIceberg support schema evolution?
PyIceberg supports schema evolution through transactional schema updates. For example, you can calltable.update_schema(), add a new column such as pickup_borough, and commit the change. Existing rows remain compatible, while new data can include the added column.Why is Polars useful for Apache Iceberg analytics?
Polars is a high-performance Python dataframe library with lazy execution and columnar optimization. It can scan Iceberg tables, push filters into the scan, perform aggregations, and collect results efficiently. Polars delegates Iceberg catalog handling to PyIceberg and is especially useful for fast analytical workloads.What role does DuckDB play in Python-based Iceberg workflows?
DuckDB provides an in-process SQL engine for querying Iceberg tables directly from Python. With the Iceberg extension, DuckDB can read Iceberg metadata files using functions such asiceberg_scan, attach Iceberg catalogs, create tables, insert data, and perform time-travel queries against specific snapshots.How does Daft integrate with Apache Iceberg?
Daft reads Iceberg tables through PyIceberg and loads them into distributed Daft DataFrames. It is optimized for large-scale analytical processing using Apache Arrow. Daft currently focuses on reading and transforming Iceberg data rather than writing directly to Iceberg tables, though processed data can be exported to Parquet and registered with PyIceberg.How can Python users interact with Iceberg tables through Dremio?
Python users can interact with Dremio using ODBC or Apache Arrow Flight. ODBC is useful for lightweight SQL access, while Arrow Flight provides faster columnar transport for large result sets. Through Dremio, users can query Iceberg tables and perform DDL and DML operations such asCREATE, INSERT, DELETE, MERGE, and UPDATE.
Architecting an Apache Iceberg Lakehouse ebook for free
