Implementation:Huggingface Datasets Parquet Builder

Knowledge Sources	Huggingface Datasets HF Datasets Docs
Domains	Data_Loading, Columnar
Last Updated	2026-02-14 18:00 GMT

Overview

Packaged dataset builder for loading Parquet files with column filtering and row group support into Arrow-backed datasets provided by the HuggingFace Datasets library.

Description

Parquet is a packaged dataset builder extending ArrowBasedBuilder that reads Apache Parquet files into Arrow tables. It is configured via ParquetConfig, a dataclass extending BuilderConfig, with fields for batch_size (defaults to row group size), columns (column subset selection), features (explicit schema), filters (predicate pushdown for row filtering), fragment_scan_options (PyArrow scan configuration for buffering and caching), and on_bad_files (error handling strategy: "error", "warn", or "skip").

During split generation, if features are not explicitly provided, the builder infers them from the Parquet file's Arrow schema using pq.read_schema(). Schema inference respects the on_bad_files policy, skipping unreadable files when configured. If columns is specified, features are filtered to include only the requested columns.

The _generate_tables method reads Parquet files using PyArrow's dataset API (ParquetFileFormat and fragments), supporting row group subsetting, column projection, and filter expression pushdown. Batch size defaults to the number of rows in the first row group if not explicitly configured. The _generate_more_gen_kwargs method enables further parallelism by splitting files into individual row groups for separate processing.

Usage

Use Parquet via load_dataset("parquet", data_files=...) to load Parquet files. It supports efficient column selection, row filtering, and configurable scan options for streaming workloads.

Code Reference

Source Location

• Repository: datasets
• File: src/datasets/packaged_modules/parquet/parquet.py
• Lines: 1-217

Signature

@dataclass
class ParquetConfig(datasets.BuilderConfig):
    batch_size: Optional[int] = None
    columns: Optional[list[str]] = None
    features: Optional[datasets.Features] = None
    filters: Optional[Union[ds.Expression, list[tuple], list[list[tuple]]]] = None
    fragment_scan_options: Optional[ds.ParquetFragmentScanOptions] = None
    on_bad_files: Literal["error", "warn", "skip"] = "error"


class Parquet(datasets.ArrowBasedBuilder):
    BUILDER_CONFIG_CLASS = ParquetConfig

    def _info(self):
    def _split_generators(self, dl_manager):
    def _cast_table(self, pa_table: pa.Table) -> pa.Table:
    def _generate_shards(self, files, row_groups_list):
    def _generate_more_gen_kwargs(self, files, row_groups_list):
    def _generate_tables(self, files, row_groups_list):

Import

from datasets.packaged_modules.parquet.parquet import Parquet, ParquetConfig

I/O Contract

Inputs (ParquetConfig)

Name	Type	Required	Description
batch_size	Optional[int]	No	Number of rows per RecordBatch. Defaults to the row group size of the first row group.
columns	Optional[list[str]]	No	List of column names to load. All columns are loaded by default.
features	Optional[Features]	No	Explicit schema to apply. If None, inferred from the Parquet file metadata.
filters	Optional[Union[Expression, list[tuple], list[list[tuple]]]]	No	Row filter expressions for predicate pushdown. Supports PyArrow Expression objects or tuple-based filter syntax.
fragment_scan_options	Optional[ParquetFragmentScanOptions]	No	PyArrow scan options for configuring buffering, caching, and prefetching behavior. Added in version 4.2.0.
on_bad_files	Literal["error", "warn", "skip"]	No	How to handle unreadable files. Defaults to "error". Added in version 4.2.0.

Outputs

Name	Type	Description
dataset	Dataset	An Arrow-backed Dataset constructed from the Parquet file contents.

Usage Examples

Basic Usage

from datasets import load_dataset

# Load Parquet files
ds = load_dataset("parquet", data_files="data/train.parquet", split="train")
print(ds[0])

Column Selection

from datasets import load_dataset

# Load only specific columns
ds = load_dataset(
    "parquet",
    data_files="data/train.parquet",
    columns=["col_0", "col_1"],
    split="train",
)
print(ds.column_names)  # ["col_0", "col_1"]

Row Filtering with Predicate Pushdown

from datasets import load_dataset

# Filter rows using tuple-based syntax
ds = load_dataset(
    "parquet",
    data_files="data/train.parquet",
    filters=[("label", "==", 1)],
    streaming=True,
)

Custom Scan Options for Streaming

import pyarrow
import pyarrow.dataset
from datasets import load_dataset

# Increase request size and enable prefetching
fragment_scan_options = pyarrow.dataset.ParquetFragmentScanOptions(
    cache_options=pyarrow.CacheOptions(
        prefetch_limit=1,
        range_size_limit=128 << 20,  # 128MiB
    ),
)
ds = load_dataset(
    "parquet",
    data_files="data/train.parquet",
    streaming=True,
    fragment_scan_options=fragment_scan_options,
)

Related Pages

Implements Principle

• Principle:Huggingface_Datasets_Parquet_Dataset_Building

Requires Environment

• Environment:Huggingface_Datasets_Python_PyArrow_Core