Principle:DataTalksClub Data engineering zoomcamp GCS Data Upload

Knowledge Sources	Google Cloud Storage Documentation
Domains	Data_Ingestion, GCS, Cloud_Storage
Last Updated	2026-02-09 00:00 GMT

Overview

A data movement pattern that transfers files from web sources or local storage to Google Cloud Storage buckets.

Description

Uploading data to cloud object storage is a foundational step in modern data pipelines. The GCS Data Upload pattern encompasses the full lifecycle of acquiring data from an external source and placing it into a cloud storage bucket where it can be consumed by downstream processing systems (data warehouses, batch processing frameworks, analytics engines).

The pattern consists of three sequential phases:

• Download: Data is fetched from a web source (HTTP/HTTPS endpoint, public dataset, API) to local or temporary storage. This phase must handle network instability, large file sizes, and varying source formats. Chunked downloading -- reading the response in fixed-size blocks rather than loading everything into memory -- is critical for large datasets.

• Transform (optional): The downloaded data may need format conversion before upload. Common transformations include converting CSV files to columnar formats (such as Parquet) for query performance, compressing data to reduce storage costs and transfer time, or partitioning a single large file into smaller segments for parallel downstream processing.

• Upload: The local file is transferred to a GCS bucket at a specified object path. The upload phase must handle authentication (via service account credentials), bucket creation or validation, and reliable transfer of potentially large files. For files exceeding a size threshold, resumable or multipart uploads are used to allow retries without re-uploading the entire file.

This pattern is commonly used as the first stage of an ELT (Extract, Load, Transform) pipeline, where raw data is landed in cloud storage before being loaded into a data warehouse for transformation.

Usage

Use this principle when:

• You need to ingest external datasets into a cloud-based data pipeline.
• Your data sources are publicly accessible web endpoints that serve CSV, Parquet, or other file formats.
• You want to decouple data acquisition from data processing by staging files in object storage.
• You need to build reproducible, automated data ingestion that can be scheduled or triggered.
• You are constructing a data lake where raw files are stored before downstream transformation.

Theoretical Basis

The upload pipeline follows a linear three-stage model:

PIPELINE: Download -> Transform -> Upload

STAGE 1 - DOWNLOAD:
    FUNCTION download_file(source_url, local_path):
        response = HTTP_GET(source_url, stream = TRUE)
        ASSERT response.status == 200

        file = OPEN(local_path, mode = "write_binary")
        FOR EACH chunk IN response.iter_chunks(size = CHUNK_SIZE):
            file.write(chunk)
        file.close()

        RETURN local_path

STAGE 2 - TRANSFORM (optional):
    FUNCTION convert_to_columnar(input_path, output_path):
        dataframe = read_tabular(input_path)

        -- Apply schema corrections:
        FOR EACH column IN dataframe.columns:
            IF column.type IS ambiguous:
                column.type = infer_or_cast(column)

        write_columnar(dataframe, output_path, compression = "gzip")
        RETURN output_path

STAGE 3 - UPLOAD:
    FUNCTION upload_to_bucket(bucket_name, object_path, local_path):
        client = create_storage_client(credentials = SERVICE_ACCOUNT)

        bucket = client.get_bucket(bucket_name)
        IF bucket DOES NOT EXIST:
            bucket = client.create_bucket(bucket_name, location = REGION)

        blob = bucket.blob(object_path)

        file_size = get_file_size(local_path)
        IF file_size > RESUMABLE_THRESHOLD:
            blob.upload_from_filename(local_path, resumable = TRUE)
        ELSE:
            blob.upload_from_filename(local_path)

        RETURN blob.public_url

Retry logic:

FUNCTION upload_with_retry(bucket, object_path, local_path, max_retries = 3):
    FOR attempt IN 1..max_retries:
        TRY:
            upload_to_bucket(bucket, object_path, local_path)
            RETURN SUCCESS
        CATCH TransientError:
            wait_time = BASE_DELAY * (2 ^ attempt)   -- exponential backoff
            SLEEP(wait_time)
    RAISE UploadFailedError("Exceeded max retries")

The key design considerations are:

1. Memory efficiency: Chunked downloading and streaming uploads avoid loading entire files into memory, enabling processing of datasets larger than available RAM.
2. Idempotency: Uploading the same file to the same object path overwrites the previous version, making the operation safe to retry without creating duplicates.
3. Format optimization: Converting row-oriented formats (CSV) to columnar formats (Parquet) before upload reduces downstream query costs and improves read performance for analytical workloads.
4. Separation of concerns: By staging data in object storage, the ingestion process is decoupled from the processing and querying systems, allowing each to scale independently.

Related Pages

• Implementation:DataTalksClub_Data_engineering_zoomcamp_Web_To_GCS_Upload
• Implementation:DataTalksClub_Data_engineering_zoomcamp_GCS_Parquet_Upload