Implementation:NVIDIA NeMo Curator JSONLReader

Knowledge Sources	NVIDIA NeMo Curator
Domains	Data Ingestion, IO, JSONL, Data Pipeline
Last Updated	2026-02-14 00:00 GMT

Overview

Provides JSONL file reading capabilities with a low-level JsonlReaderStage and a high-level JsonlReader composite stage for ingesting JSONL data into the NeMo Curator pipeline.

Description

This module contains two classes that implement JSONL file reading at different abstraction levels:

JsonlReaderStage extends BaseReader and implements the read_data abstract method. It reads each JSONL file path using pd.read_json with lines=True enforced (raising a ValueError if lines=False is explicitly passed). After reading each file, it optionally selects specific columns using pandas_select_columns, then concatenates all resulting DataFrames with pd.concat. If no data is read from any file, it raises a ValueError.

JsonlReader is a CompositeStage[_EmptyTask, DocumentBatch] dataclass that provides a high-level interface for reading JSONL files. It decomposes into two stages:

1. FilePartitioningStage - discovers and groups files into partitions based on files_per_partition or blocksize
2. JsonlReaderStage - reads each file group into a DocumentBatch

This two-level design separates file discovery and partitioning from the actual reading, enabling better parallelism in distributed execution.

Usage

Use JsonlReader as the primary entry point for reading JSONL files in a pipeline. Use JsonlReaderStage directly when you already have FileGroupTask inputs from an upstream FilePartitioningStage or custom file grouping logic.

Code Reference

Source Location

• Repository: NeMo-Curator
• File: nemo_curator/stages/text/io/reader/jsonl.py
• Lines: 1-146

Signature

@dataclass
class JsonlReaderStage(BaseReader):
    name: str = "jsonl_reader"

    def read_data(
        self,
        paths: list[str],
        read_kwargs: dict[str, Any] | None = None,
        fields: list[str] | None = None,
    ) -> pd.DataFrame | None: ...


@dataclass
class JsonlReader(CompositeStage[_EmptyTask, DocumentBatch]):
    file_paths: str | list[str]
    files_per_partition: int | None = None
    blocksize: int | str | None = None
    fields: list[str] | None = None
    read_kwargs: dict[str, Any] | None = None
    task_type: Literal["document", "image", "video", "audio"] = "document"
    file_extensions: list[str] = field(default_factory=...)
    _generate_ids: bool = False
    _assign_ids: bool = False
    name: str = "jsonl_reader"

Import

from nemo_curator.stages.text.io.reader.jsonl import JsonlReaderStage, JsonlReader

I/O Contract

Inputs (JsonlReader)

Name	Type	Required	Description
file_paths	str or list[str]	Yes	Path or list of paths to JSONL files or directories containing JSONL files
files_per_partition	int or None	No	Number of files to group per partition (default: None)
blocksize	int or str or None	No	Target block size for file partitioning, e.g. "128MB" (default: None)
fields	list[str] or None	No	If specified, only read these columns from the JSONL files (default: None)
read_kwargs	dict[str, Any] or None	No	Additional keyword arguments passed to pd.read_json (default: None)
task_type	Literal["document", ...]	No	Type of task; only "document" is currently supported (default: "document")
file_extensions	list[str]	No	File extensions to match when discovering files (default: JSONL extensions)
_generate_ids	bool	No	Whether to generate monotonically increasing deduplication IDs (default: False)
_assign_ids	bool	No	Whether to assign pre-computed deduplication IDs (default: False)

Inputs (JsonlReaderStage)

Name	Type	Required	Description
FileGroupTask	FileGroupTask	Yes	Task containing a list of JSONL file paths to read

Outputs

Name	Type	Description
DocumentBatch	DocumentBatch	Contains the concatenated data from all JSONL files in the group as a Pandas DataFrame

Usage Examples

Basic Usage with JsonlReader

from nemo_curator.stages.text.io.reader.jsonl import JsonlReader

# Read JSONL files from a directory
reader = JsonlReader(
    file_paths="/data/corpus/",
    files_per_partition=10,
    fields=["text", "url", "language"],
)

With Custom Read Settings

from nemo_curator.stages.text.io.reader.jsonl import JsonlReader

reader = JsonlReader(
    file_paths=["/data/part1.jsonl", "/data/part2.jsonl"],
    blocksize="256MB",
    read_kwargs={
        "dtype": {"score": float},
        "storage_options": {"anon": True},
    },
)

Using JsonlReaderStage Directly

from nemo_curator.stages.text.io.reader.jsonl import JsonlReaderStage

reader_stage = JsonlReaderStage(
    fields=["text", "title"],
    read_kwargs={"encoding": "utf-8"},
    _generate_ids=True,
)

Implementation Details

Lines Enforcement

JsonlReaderStage always forces lines=True when calling pd.read_json. If lines=False is explicitly passed in read_kwargs, a ValueError is raised because line-delimited reading is fundamental to the JSONL format.

Composite Decomposition

JsonlReader.decompose() creates a two-stage pipeline:

1. FilePartitioningStage handles file discovery, globbing, and grouping using file_extensions and partitioning parameters
2. JsonlReaderStage receives FileGroupTask inputs and reads the actual JSONL data

Storage options from read_kwargs are propagated to both the FilePartitioningStage (for remote filesystem access during file discovery) and the JsonlReaderStage (for reading).

Task Type Restriction

Currently, only task_type="document" is supported. Passing other values ("image", "video", "audio") raises a NotImplementedError.

Related Pages

• NVIDIA_NeMo_Curator_BaseReader - Abstract base class that JsonlReaderStage extends
• NVIDIA_NeMo_Curator_ParquetReader - Analogous reader for Parquet format files
• NVIDIA_NeMo_Curator_BaseWriter - Base class for the corresponding writer stages
• NVIDIA_NeMo_Curator_FilePartitioningStage - Stage that handles file discovery and grouping
• Environment:NVIDIA_NeMo_Curator_Python_Linux_Base