Implementation:Datajuicer Data juicer TsvFormatter

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Loading, Formatting
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for loading and formatting TSV (tab-separated values) files as datasets provided by Data-Juicer.

Description

TsvFormatter extends LocalFormatter with SUFFIXES = ['.tsv'] and type='csv' with delimiter='\t', reusing the CSV reader with a tab delimiter. It is registered with the FORMATTERS registry via @FORMATTERS.register_module(). This enables Data-Juicer to load TSV-formatted datasets, handling the tab-delimited variant of CSV without requiring users to manually specify the delimiter.

Usage

Use when loading TSV-formatted datasets into Data-Juicer for processing, either through direct instantiation or through the automatic format detection system.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/format/tsv_formatter.py

Signature

@FORMATTERS.register_module()
class TsvFormatter(LocalFormatter):
    SUFFIXES = [".tsv"]

    def __init__(self, dataset_path, suffixes=None, **kwargs):

Import

from data_juicer.format.tsv_formatter import TsvFormatter

I/O Contract

Inputs

Name	Type	Required	Description
dataset_path	str	Yes	Path to a TSV dataset file or directory containing TSV files
suffixes	list	No	File suffixes to be processed. Default: ['.tsv']
**kwargs	Any	No	Extra arguments passed to the parent LocalFormatter (note: delimiter='\t' is set automatically)

Outputs

Name	Type	Description
dataset	Dataset	A unified HuggingFace Dataset loaded from the TSV files

Usage Examples

from data_juicer.format.tsv_formatter import TsvFormatter

# Load a TSV dataset
formatter = TsvFormatter(dataset_path="/path/to/data.tsv")
dataset = formatter.load_dataset(num_proc=4)

# Load from a directory of TSV files
formatter = TsvFormatter(dataset_path="/path/to/tsv_dir/")
dataset = formatter.load_dataset()

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment