Implementation:Datajuicer Data juicer CsvFormatter

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Loading, Formatting
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for loading and formatting CSV files as datasets provided by Data-Juicer.

Description

CsvFormatter extends LocalFormatter with SUFFIXES = ['.csv'] and type='csv', delegating all loading logic to the parent class which uses HuggingFace's load_dataset with the CSV reader. It is registered with the FORMATTERS registry via the @FORMATTERS.register_module() decorator, enabling automatic format detection.

Usage

Use when loading CSV-formatted datasets into Data-Juicer for processing, either through direct instantiation or through the automatic format detection system via load_formatter.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/format/csv_formatter.py

Signature

@FORMATTERS.register_module()
class CsvFormatter(LocalFormatter):
    SUFFIXES = [".csv"]

    def __init__(self, dataset_path, suffixes=None, **kwargs):

Import

from data_juicer.format.csv_formatter import CsvFormatter

I/O Contract

Inputs

Name	Type	Required	Description
dataset_path	str	Yes	Path to a CSV dataset file or directory containing CSV files
suffixes	list	No	File suffixes to be processed. Default: ['.csv']
**kwargs	Any	No	Extra arguments passed to the parent LocalFormatter

Outputs

Name	Type	Description
dataset	Dataset	A unified HuggingFace Dataset loaded from the CSV files

Usage Examples

from data_juicer.format.csv_formatter import CsvFormatter

# Load a CSV dataset
formatter = CsvFormatter(dataset_path="/path/to/data.csv")
dataset = formatter.load_dataset(num_proc=4)

# Load from a directory of CSV files
formatter = CsvFormatter(dataset_path="/path/to/csv_dir/")
dataset = formatter.load_dataset()

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment