Principle:Eventual Inc Daft Data Ingestion CSV

Knowledge Sources	Daft Daft Docs
Domains	Data_Engineering, Analytics
Last Updated	2026-02-08 00:00 GMT

Overview

Technique for creating lazy DataFrame scans from CSV (Comma-Separated Values) files.

Description

CSV is one of the most common text-based tabular data formats, widely used for data exchange between systems. Daft reads CSV files lazily with automatic schema inference, meaning the scan plan is constructed without reading the entire file upfront.

Key capabilities of Daft's CSV ingestion include:

• Schema inference: Column types are automatically detected by sampling the CSV data, or a user-provided schema can be used instead.
• Custom delimiters: Support for non-comma delimiters (e.g., tab-separated, pipe-separated) via the delimiter parameter.
• Quote and escape handling: Configurable quote characters, double-quote escaping, and custom escape characters for handling embedded delimiters within field values.
• Variable column support: Optionally handle rows with different numbers of columns by padding with nulls or ignoring extra columns.
• Remote storage: Support for local paths, S3, GCS, and Azure Blob Storage with glob pattern matching.
• Hive partitioning: Ability to infer partition columns from directory structures in the file path.

Usage

Use this technique when loading tabular text data from CSV files. Common scenarios include:

• Ingesting exported data from databases, spreadsheets, or legacy systems
• Loading log files and text-based data feeds
• Reading data from systems that do not support columnar formats

Theoretical Basis

CSV parsing follows a row-based text format parsing model with the following principles:

1. Sequential record parsing: Each line represents a record, with fields separated by a delimiter character. Quote characters allow delimiters and newlines within field values.
2. Schema inference via sampling: Since CSV files lack embedded type metadata, the reader samples a portion of the data to infer column types (integer, float, string, boolean, date, etc.).
3. Lazy evaluation: Like Parquet ingestion, the scan plan is constructed without reading data, enabling downstream optimizations before I/O occurs.
4. Fault tolerance: Configurable error handling for malformed rows, variable column counts, and encoding issues.

Pseudocode:
1. Parse path(s) and resolve glob patterns
2. Configure CSV parsing options (delimiter, quote, headers, etc.)
3. If infer_schema=True:
   a. Sample rows from the CSV to detect column types
4. Construct lazy scan plan with:
   - CSV source config (delimiter, quoting, escaping)
   - Storage config (IO config)
5. Apply hive partitioning if enabled
6. Return lazy DataFrame (no full data read yet)

Related Pages

Implemented By

• Implementation:Eventual_Inc_Daft_Read_Csv