Implementation:Huggingface Datasets Csv Builder

Knowledge Sources	Huggingface Datasets HF Datasets Docs
Domains	Data_Loading, Tabular
Last Updated	2026-02-14 18:00 GMT

Overview

Packaged dataset builder for loading CSV files with extensive pandas configuration, provided by the HuggingFace Datasets library.

Description

Csv is a packaged dataset builder extending datasets.ArrowBasedBuilder that loads CSV files using pandas.read_csv and converts the resulting DataFrames into Apache Arrow tables. It is paired with CsvConfig, a BuilderConfig subclass exposing approximately 37 parameters that mirror the full pandas.read_csv API, including separator, header handling, column selection, type conversion, NA handling, quoting, encoding, and chunking options.

CsvConfig includes a pd_read_csv_kwargs property that dynamically assembles the keyword arguments for pandas.read_csv, handling pandas version compatibility by conditionally removing deprecated parameters (e.g., warn_bad_lines, error_bad_lines) and parameters introduced in newer versions (e.g., encoding_errors from pandas 1.3, date_format from pandas 2.0, verbose deprecated in pandas 2.2).

The Csv builder reads files in chunks (default 10,000 rows via chunksize), yielding each chunk as a PyArrow table. When explicit features are specified, it applies schema casting via _cast_table, supporting both cheap column reordering and more expensive type coercion (e.g., string to Audio). The builder enables on-the-fly extraction for compressed files via dl_manager.download_config.extract_on_the_fly = True.

Usage

Use this builder via load_dataset("csv", data_files=...) to load CSV or TSV files. It is also triggered automatically when files with .csv or .tsv extensions are detected by the dataset loading pipeline.

Code Reference

Source Location

• Repository: datasets
• File: src/datasets/packaged_modules/csv/csv.py
• Lines: 1-206

Signature

@dataclass
class CsvConfig(datasets.BuilderConfig):
    """BuilderConfig for CSV."""
    sep: str = ","
    delimiter: Optional[str] = None
    header: Optional[Union[int, list[int], str]] = "infer"
    names: Optional[list[str]] = None
    column_names: Optional[list[str]] = None
    index_col: Optional[Union[int, str, list[int], list[str]]] = None
    usecols: Optional[Union[list[int], list[str]]] = None
    prefix: Optional[str] = None
    mangle_dupe_cols: bool = True
    engine: Optional[Literal["c", "python", "pyarrow"]] = None
    converters: dict[Union[int, str], Callable[[Any], Any]] = None
    true_values: Optional[list] = None
    false_values: Optional[list] = None
    skipinitialspace: bool = False
    skiprows: Optional[Union[int, list[int]]] = None
    nrows: Optional[int] = None
    na_values: Optional[Union[str, list[str]]] = None
    keep_default_na: bool = True
    na_filter: bool = True
    verbose: bool = False
    skip_blank_lines: bool = True
    thousands: Optional[str] = None
    decimal: str = "."
    lineterminator: Optional[str] = None
    quotechar: str = '"'
    quoting: int = 0
    escapechar: Optional[str] = None
    comment: Optional[str] = None
    encoding: Optional[str] = None
    dialect: Optional[str] = None
    error_bad_lines: bool = True
    warn_bad_lines: bool = True
    skipfooter: int = 0
    doublequote: bool = True
    memory_map: bool = False
    float_precision: Optional[str] = None
    chunksize: int = 10_000
    features: Optional[datasets.Features] = None
    encoding_errors: Optional[str] = "strict"
    on_bad_lines: Literal["error", "warn", "skip"] = "error"
    date_format: Optional[str] = None

class Csv(datasets.ArrowBasedBuilder):
    BUILDER_CONFIG_CLASS = CsvConfig

Key methods:

def _info(self):
    return datasets.DatasetInfo(features=self.config.features)

def _split_generators(self, dl_manager):
    # Downloads files, enables on-the-fly extraction
    # Returns SplitGenerator for each split with file iterables

def _cast_table(self, pa_table: pa.Table) -> pa.Table:
    # Casts table to match explicit features schema
    # Uses cheap cast when possible, expensive table_cast for type coercion

def _generate_tables(self, base_files, files_iterables):
    # Uses pd.read_csv with iterator=True and configured chunksize
    # Yields (Key, pa.Table) for each chunk from each file

Import

# Used via load_dataset
from datasets import load_dataset
ds = load_dataset("csv", data_files="path/to/file.csv")

I/O Contract

Inputs

Name	Type	Required	Description
data_files	str, list, or dict	Yes	Path(s) to CSV files. Can be a single path, a list, or a dict mapping split names to file paths.
sep	str	No	Field delimiter. Defaults to ",".
delimiter	Optional[str]	No	Alias for sep. Overrides sep if provided.
header	Optional[Union[int, list[int], str]]	No	Row number(s) to use as column names. Defaults to "infer".
names / column_names	Optional[list[str]]	No	Explicit list of column names.
features	Optional[Features]	No	Explicit feature schema for type casting.
chunksize	int	No	Number of rows per chunk. Defaults to 10000.
encoding	Optional[str]	No	File encoding (e.g. "utf-8", "latin-1").
on_bad_lines	Literal["error", "warn", "skip"]	No	How to handle bad lines. Defaults to "error".

(Plus approximately 28 additional pandas.read_csv parameters -- see CsvConfig for the full list.)

Outputs

Name	Type	Description
(from _generate_tables)	tuple[Key, pa.Table]	Yields tuples of (Key(shard_idx, batch_idx), pa_table) for each chunk from each CSV file.
(from load_dataset)	Dataset or DatasetDict	The loaded dataset with Arrow-backed storage.

Usage Examples

Basic Usage

from datasets import load_dataset

# Load a CSV file
ds = load_dataset("csv", data_files="data/train.csv", split="train")
print(ds[0])

# Load a TSV file
ds = load_dataset("csv", data_files="data/train.tsv", sep="\t", split="train")

Advanced Configuration

from datasets import load_dataset, Features, Value

# Load with explicit features, custom encoding, and skipped rows
features = Features({
    "id": Value("int64"),
    "text": Value("string"),
    "score": Value("float32"),
})
ds = load_dataset(
    "csv",
    data_files="data/train.csv",
    features=features,
    encoding="latin-1",
    skiprows=1,
    na_values=["N/A", "null"],
    split="train",
)

Multiple Splits

from datasets import load_dataset

ds = load_dataset("csv", data_files={
    "train": "data/train.csv",
    "validation": "data/val.csv",
    "test": "data/test.csv",
})
print(ds["train"].num_rows)

Related Pages

Implements Principle

• Principle:Huggingface_Datasets_CSV_Dataset_Building