4 Selecting the storage layer

The chapter explains that the storage layer is the foundation of an Apache Iceberg lakehouse because it directly affects reliability, scalability, performance, security, cost, and long-term flexibility. Rather than choosing a technology based on vendor preference, teams should begin with requirements gathered from a platform audit: how quickly data must be read and written, how many users and engines will access it concurrently, how sensitive the data is, how recoverable it must be, and what operational model the organization can sustain. The chapter emphasizes that performance should be evaluated through real workload testing, including latency, throughput, concurrency, and access patterns, while security must include access control, encryption, auditability, identity integration, and data residency. Integrity requirements include durability, replication, snapshots, versioning, checksums, recovery, and high availability, while cost planning must account for capacity, egress, requests, replication, scaling, maintenance, and staffing.

A major theme is the contrast between block storage and object storage. Block storage, used historically with systems like HDFS, offers low latency and strong performance when compute and storage are closely coupled, making it useful for legacy Hadoop environments and certain high-performance on-premises workloads. However, it is harder to scale independently, requires more operational management, and lacks many cloud-native features. Object storage, by contrast, stores data as metadata-rich objects accessed through APIs, most commonly the S3 API. It supports separation of compute and storage, elastic scaling, strong durability, lifecycle management, encryption, and broad compatibility, which makes it the dominant model for modern Iceberg deployments. The chapter also highlights two key standards: Apache Parquet, whose columnar layout, compression, statistics, and schema awareness make it ideal for Iceberg analytics, and the S3 API, whose widespread adoption enables tools and engines to interact consistently with object storage across cloud, on-premises, and hybrid environments.

The chapter surveys storage options including HDFS, Amazon S3, Google Cloud Storage, Azure Blob Storage and ADLS, MinIO, Ceph, NetApp StorageGRID, Pure Storage, Dell ECS, and Wasabi, showing that each fits different organizational priorities. Cloud-native stores such as Amazon S3, Google Cloud Storage, and Azure services offer managed scalability, strong ecosystem integration, and enterprise security features, while on-premises and hybrid options such as MinIO, Ceph, StorageGRID, Pure Storage, and Dell ECS provide more control over residency, compliance, performance, or infrastructure. Wasabi is presented as a simpler, cost-predictable S3-compatible option, while HDFS remains useful mainly for organizations modernizing existing Hadoop investments. The chapter concludes that storage selection should be driven by requirements rather than a universal best choice: high-concurrency dashboards, batch ETL, machine learning pipelines, regulated archives, healthcare data, geo-distributed recovery, edge deployments, startups, enterprises, research institutions, and government agencies may all prioritize different trade-offs. Iceberg’s modular design supports this flexibility, but the chosen storage layer must align with workload needs, governance obligations, operational maturity, and long-term strategy.

The factors to consider when understanding a storage layers performance.

The factors to consider when analyzing a storage layers security.

The integrity factors to consider when assessing a storage system.

Factors when considering the cost of a storage system.

Illustrating how a single file is stored across multiple blocks in a block storage system.

A file is stored as a blob (binary large object) with an Object with metadata and id used to access it, user requests the file by key and the system then uses the metadata to locate and retrieve the file.

The engine doing an Iceberg transaction will use the S3 API to talk to an object storage platform where it will store data as a parquet file along with the Iceberg metadata.

Based on the metadata in a parquet row group footers, we can determine whether a particular row group needs to be scanned or can be skipped for faster reading.

Each write to the table introduces new files, while the metadata of each snapshot tracks the files in that table at that time.

Different engines working with the same Iceberg table can individually request files concurrently from the object storage system.

• The storage layer is a foundational component of any Apache Iceberg lakehouse, influencing performance, scalability, cost, and compliance.
• Before selecting a storage solution, it’s critical to define requirements based on file retrieval performance, security, data integrity, and operational costs, ideally as part of a structured audit.
• Storage systems generally fall into two categories: block storage, which offers low-latency performance but higher operational complexity, and object storage, which is more scalable and cloud-native.
• Apache Parquet is the default file format for Iceberg due to its columnar design and rich metadata support, enabling efficient pruning and high-performance queries.
• The S3 API has become the industry standard for object storage access, enabling broad interoperability across cloud and on-premises systems.
• A wide range of storage solutions support Iceberg deployments, including cloud services like Amazon S3, Google Cloud Storage, and Azure Blob/ADLS, as well as on-premises options such as MinIO, Ceph, NetApp StorageGRID, Dell ECS, Pure Storage, and Wasabi.
• Each storage platform offers trade-offs across cost, complexity, performance, and governance. No single solution fits all use cases.
• Real-world requirements, like interactive latency, legal compliance, disaster recovery, or budget constraints, should guide final selection. Hypothetical examples illustrate how different needs might map to different technologies.
• By aligning your storage layer to the specific requirements uncovered during your audit, you ensure that your Iceberg lakehouse is not only technically sound, but also operationally sustainable and aligned with business goals.

FAQ