Principle:DataTalksClub Data engineering zoomcamp Chunked Data Ingestion

Metadata
Knowledge Sources	DataTalksClub/data-engineering-zoomcamp
Domains	Data Ingestion, Memory Management, Batch Processing, ETL
Last Updated	2026-02-09 14:00 GMT

Overview

Processing large datasets in fixed-size chunks via an iterator pattern to control memory usage, where the first chunk establishes the target schema and subsequent chunks append data.

Description

Large CSV files (hundreds of megabytes to several gigabytes) cannot always be loaded entirely into memory, especially in constrained environments like Docker containers with limited RAM. The chunked ingestion principle addresses this by reading and processing data in fixed-size batches.

The core mechanism works as follows:

Iterator creation: Instead of reading the entire file at once, the reader is configured to return an iterator. Each call to the iterator yields a fixed number of rows (the chunk size).
Schema creation from first chunk: The first chunk is used to create the target database table with the correct column names and types. Only the schema (zero rows) is written with a replace strategy, ensuring any previous table of the same name is dropped.
Append for subsequent chunks: All chunks (including the first) are inserted into the table using an append strategy. This means each chunk adds its rows to the existing table without dropping or recreating it.
Progress tracking: Since the total number of chunks is unknown in advance (the iterator is lazy), a progress indicator tracks how many chunks have been processed so far.

The chunk size is a tunable parameter that balances two concerns:

Too small: Increases the number of database round-trips and per-chunk overhead, reducing throughput.
Too large: Increases peak memory usage, potentially causing out-of-memory errors.

A typical starting value is 100,000 rows per chunk, which provides a good balance for most tabular datasets.

Usage

Use this principle when:

The source dataset is too large to fit in memory.
You are loading data into a database and want to see incremental progress.
You need to control peak memory usage in a resource-constrained environment.
The target table schema should be derived from the data itself rather than defined separately.

Theoretical Basis

The chunked ingestion pattern follows a producer-consumer model with lazy evaluation:

DEFINE chunk_iterator = open_csv_as_iterator(source, chunk_size=N):
    WHILE source has more rows:
        READ next N rows into chunk
        YIELD chunk

DEFINE ingest(chunk_iterator, target_table, database_engine):
    is_first = TRUE

    FOR each chunk IN chunk_iterator:
        IF is_first:
            CREATE table target_table with schema from chunk (zero rows)
            SET is_first = FALSE

        INSERT chunk rows into target_table (append mode)
        REPORT progress (chunks processed so far)

The important architectural insight is the separation of schema creation from data insertion. By using the first chunk's column types to create the table, the pipeline ensures that the database schema exactly matches the data types produced by the type configuration. The replace on schema creation guarantees idempotency: running the pipeline twice produces the same result, with the second run fully replacing the first.

The iterator pattern also enables streaming from remote sources: the CSV reader can fetch data directly from a URL, decompressing gzip-compressed content on the fly without ever writing the full file to disk.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment