Principle:Eventual Inc Daft Iceberg Reading
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Data_Lakehouse |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Iceberg reading is the technique for creating a lazy DataFrame scan of an Apache Iceberg table with snapshot isolation, supporting time travel and predicate pushdown.
Description
Iceberg reading creates a lazy DataFrame scan of an Apache Iceberg table. The scan leverages Iceberg's metadata layer for partition pruning and predicate pushdown, meaning only the data files relevant to applied filters are actually read. Time travel is supported through snapshot IDs, allowing queries against historical versions of the table. The resulting DataFrame is lazy: no data is read until an action (such as show() or collect()) triggers execution. This enables the query optimizer to push filters and projections down to the storage layer for efficient data retrieval.
Usage
Use Iceberg reading when you need to read data from an Apache Iceberg table with transactional guarantees. This is appropriate for analytical workloads against data lakes that use Iceberg as their table format, especially when you need consistent reads via snapshot isolation or need to query historical data via time travel.
Theoretical Basis
Apache Iceberg is an open table format that provides snapshot isolation for analytical reads. Key concepts:
- Snapshots: Each write to an Iceberg table creates an immutable snapshot, enabling time travel and consistent reads.
- Manifest lists and manifests: Metadata files that track which data files belong to each snapshot, enabling partition pruning without listing storage directories.
- Hidden partitioning: Partition transforms are stored in metadata, allowing the engine to prune partitions automatically without user-specified partition filters.
- Schema evolution: The table schema can evolve over time while maintaining backward compatibility with existing data files.
1. Resolve the Iceberg table (from PyIceberg object or metadata path)
2. Select the target snapshot (latest or specified snapshot_id)
3. Read the snapshot's manifest list to identify relevant data files
4. Apply partition pruning and predicate pushdown via Iceberg metadata
5. Create a lazy scan operator that reads only the required data files on execution