Implementation:Datajuicer Data juicer Load Formatter
| Knowledge Sources | |
|---|---|
| Domains | Data_Loading, Formatting |
| Last Updated | 2026-02-14 16:00 GMT |
Overview
Concrete tool for automatic format detection and formatter selection provided by Data-Juicer.
Description
load_formatter is a factory function that automatically selects and instantiates the most appropriate formatter based on the file extensions found at the given dataset path. It scans the dataset path for files, groups them by extension using find_files_with_suffix, then scores each registered formatter in the FORMATTERS registry by how many matching files its SUFFIXES cover. The formatter with the highest match count is selected, instantiated with the intersecting suffixes, and returned. This is the auto-detection entry point for the format system.
Usage
Use when you want Data-Juicer to automatically detect and load datasets without manually specifying the file format, such as in the DefaultLocalDataLoadStrategy and other high-level callers.
Code Reference
Source Location
- Repository: Datajuicer_Data_juicer
- File:
data_juicer/format/load.py
Signature
def load_formatter(
dataset_path,
text_keys=None,
suffixes=None,
add_suffix=False,
**kwargs
) -> BaseFormatter:
Import
from data_juicer.format.load import load_formatter
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| dataset_path | str | Yes | Path to a dataset file or directory to auto-detect format from |
| text_keys | List[str] | No | Key names of fields that store sample text. Default: None |
| suffixes | list | No | File suffixes to filter by. Default: None (auto-detect all) |
| add_suffix | bool | No | Whether to add file suffix to dataset meta info. Default: False |
| **kwargs | Any | No | Extra arguments passed to the selected formatter |
Outputs
| Name | Type | Description |
|---|---|---|
| formatter | BaseFormatter | An instantiated formatter matching the detected file format (e.g. JsonFormatter, CsvFormatter) |
Usage Examples
from data_juicer.format.load import load_formatter
# Auto-detect format and load
formatter = load_formatter(
dataset_path="/path/to/dataset/",
text_keys=["text"]
)
dataset = formatter.load_dataset(num_proc=4)
# Auto-detect with suffix filtering
formatter = load_formatter(
dataset_path="/path/to/mixed_data/",
suffixes=[".json", ".jsonl"]
)
dataset = formatter.load_dataset()