7 Designing the federation layer

A federation layer lets users query data across multiple systems without physically moving all of it into Apache Iceberg tables. It acts as a bridge between the lakehouse and external systems such as SaaS platforms, legacy databases, operational systems, cloud warehouses, object storage, and third-party data sources. This allows teams to enrich Iceberg-based analytics with just-in-time access to data wherever it lives, while optionally providing a semantic layer for consistent business logic and governance.

Not all data is worth ingesting into the lakehouse. Some datasets are short-lived, rarely accessed, expensive to replicate, governed by external systems, or too operationally complex to move. Federation avoids unnecessary duplication, storage cost, ETL overhead, and latency by allowing data to be queried in place. It is especially useful for exploratory analysis, SaaS data, operational systems, hybrid environments, and gradual cloud migrations.

Common use cases include joining historical Iceberg data with live operational data, querying SaaS or third-party systems directly, supporting exploratory analysis before building ETL pipelines, enabling hybrid or multi-cloud analytics, and reducing duplication of infrequently used or high-volume datasets. For example, a fraud detection team might join user behavior logs stored in Iceberg with real-time payment activity from an OLTP database.

Federation can introduce performance, governance, and operational challenges. Remote systems may have different latency profiles, limited concurrency, inconsistent schemas, or data freshness differences. Federated queries can also place unexpected analytical workloads on systems not designed for them, potentially causing degradation or outages. To address these risks, federation platforms need pushdown optimization, workload management, access controls, semantic modeling, and careful source-system governance.

A strong federation layer should meet three major requirements. First, it must connect securely to diverse external data sources without requiring duplication. Second, it should support a semantic layer for reusable metrics, governed definitions, and consistent business logic. Third, it should provide seamless connectivity to analytics tools through interfaces such as JDBC, ODBC, REST APIs, Arrow Flight, ADBC, and emerging AI-oriented protocols such as MCP.

A semantic layer standardizes business definitions across federated and lakehouse data. It allows teams to define metrics, dimensions, calculations, joins, and governed views once, then reuse them across BI tools, notebooks, applications, and AI agents. This prevents different departments from calculating important measures, such as revenue or churn, in conflicting ways. In a federated architecture, the semantic layer also hides source-system complexity and gives users a stable, trusted interface.

Dremio is an Iceberg-focused lakehouse query platform with a built-in semantic layer, native Iceberg catalog capabilities, reflections for query acceleration, caching, governance features, and user-friendly self-service experiences. Trino is a general-purpose distributed SQL engine known for broad connector support, a modular plugin architecture, and strong federated query flexibility. Dremio is often a strong fit for governed Iceberg lakehouses and business-user enablement, while Trino is often favored in highly fragmented environments where connector breadth and customization are top priorities.

Dremio uses Apache Arrow for columnar in-memory execution, Data Reflections for transparent query acceleration, Cloud Columnar Cache for reducing object-storage reads, runtime filtering to minimize data movement, and isolated execution engines for workload management. These features help reduce latency and improve responsiveness when querying Iceberg tables, raw files, databases, and other federated sources.

Trino is highly modular and connector-rich. It supports many data lake formats, databases, warehouses, NoSQL stores, streaming systems, cloud platforms, and analytical systems. Its coordinator-worker architecture breaks queries into stages, tasks, and splits that run in parallel across workers. Trino also supports pushdown, dynamic filtering, broadcast joins, partitioned joins, custom connectors through its SPI, and integration with external security systems. This makes it useful for large, heterogeneous, multi-cloud, or on-premises environments.

The choice depends on use case, data landscape, team skills, governance needs, and deployment preferences. Dremio is a strong fit when Apache Iceberg is central, when a built-in semantic layer and governed self-service are important, or when hybrid Iceberg lakehouse modernization is the goal. Trino is a strong fit when broad connector coverage, mixed table formats such as Hudi or Delta Lake, and deep customization are priorities. Alternatives such as OneLake shortcuts may suit storage virtualization and predictable access patterns, while Spice.ai may fit AI-native applications requiring cached views, SQL access, vector search, and model orchestration.