Implementation:NVIDIA NeMo Curator Modify Module

Knowledge Sources	NVIDIA NeMo Curator
Domains	Data Curation, Text Processing, Pipeline Stages
Last Updated	2026-02-14 00:00 GMT

Overview

The Modify stage applies one or more DocumentModifier instances or callable functions to document fields, supporting single-input in-place modification and multi-input transformations.

Description

Modify is a dataclass-based ProcessingStage that bridges the DocumentModifier abstraction with the pipeline system. All text modifications (URL removal, markdown stripping, Unicode fixing, and similar operations) flow through this stage.

The stage accepts three main parameters:

modifier_fn: A single DocumentModifier, a callable, or a list mixing both. When a DocumentModifier is provided, its modify_document method is called. When a plain callable is provided, it is invoked directly.

input_fields: Supports three forms: a single string (reused for all modifiers), a list of strings (one field per modifier), or a list of lists (per-modifier multiple input fields). For single-input modifiers, the function is applied via df[col].apply(). For multi-input modifiers, dictionaries of column values are passed to the function.

output_fields: When None and all modifiers have exactly one input field, results are written in-place to the input column. When any modifier has multiple inputs, output_fields is required. Can be a single string (replicated for all modifiers) or a list.

During __post_init__, the stage normalizes all inputs into parallel lists using helper functions: _validate_and_normalize_modifiers, _normalize_input_fields, and _normalize_output_fields. Extensive validation ensures consistent counts between modifiers, input fields, and output fields. The stage name is auto-derived from modifier names using _get_modifier_stage_name.

Usage

Use Modify when you need to apply text transformation operations to document fields within a pipeline. This includes applying built-in DocumentModifier subclasses (such as URL removers or Unicode fixers) or custom callables for ad-hoc text transformations. Multiple modifiers can be chained in a single stage invocation.

Code Reference

Source Location

Repository: NeMo-Curator
File: nemo_curator/stages/text/modules/modifier.py
Lines: 1-222

Signature

@dataclass
class Modify(ProcessingStage[DocumentBatch, DocumentBatch]):
    modifier_fn: Callable | DocumentModifier | list[DocumentModifier | Callable]
    input_fields: str | list[str] | list[list[str]] = "text"
    output_fields: str | list[str | None] | None = None
    name: str = "modifier_fn"

    def __post_init__(self): ...
    def inputs(self) -> tuple[list[str], list[str]]: ...
    def outputs(self) -> tuple[list[str], list[str]]: ...
    def process(self, batch: DocumentBatch) -> DocumentBatch | None: ...

Helper Functions

def _modifier_name(x: DocumentModifier | Callable) -> str: ...
def _get_modifier_stage_name(modifiers: list[DocumentModifier | Callable]) -> str: ...
def _validate_and_normalize_modifiers(...) -> list[DocumentModifier | Callable]: ...
def _normalize_input_fields(...) -> list[list[str]]: ...
def _normalize_output_fields(...) -> list[str]: ...

Import

from nemo_curator.stages.text.modules.modifier import Modify

I/O Contract

Inputs

Name	Type	Required	Description
modifier_fn	Callable, DocumentModifier, or list	Yes	The modifier(s) to apply to the data
input_fields	str, list[str], or list[list[str]]	No	Input column name(s); defaults to `"text"`
output_fields	str, list[str or None], or None	No	Output column name(s); None means in-place modification
batch	DocumentBatch	Yes	The input batch of documents to modify

Outputs

Name	Type	Description
DocumentBatch	DocumentBatch	A new batch with modified fields applied to the DataFrame

Usage Examples

Single Modifier In-Place

from nemo_curator.stages.text.modules.modifier import Modify

# Apply a simple callable to the "text" column in-place
modify_stage = Modify(modifier_fn=str.lower, input_fields="text")
output_batch = modify_stage.process(input_batch)

Using a DocumentModifier

from nemo_curator.stages.text.modules.modifier import Modify
from nemo_curator.stages.text.modifiers.doc_modifier import DocumentModifier

class StripWhitespace(DocumentModifier):
    name = "strip_whitespace"
    def modify_document(self, text: str) -> str:
        return text.strip()

modify_stage = Modify(modifier_fn=StripWhitespace())
output_batch = modify_stage.process(input_batch)

Chaining Multiple Modifiers

from nemo_curator.stages.text.modules.modifier import Modify

# Chain multiple modifiers applied to the same column
modify_stage = Modify(
    modifier_fn=[str.lower, str.strip],
    input_fields="text",
)
output_batch = modify_stage.process(input_batch)

Multi-Input Modifier with Explicit Output

from nemo_curator.stages.text.modules.modifier import Modify

def combine_fields(title: str, body: str) -> str:
    return f"{title}\n\n{body}"

modify_stage = Modify(
    modifier_fn=combine_fields,
    input_fields=[["title", "body"]],
    output_fields="combined_text",
)
output_batch = modify_stage.process(input_batch)

Related Pages

Environment:NVIDIA_NeMo_Curator_Python_Linux_Base
NVIDIA_NeMo_Curator_Modify - Higher-level Modify pipeline integration
NVIDIA_NeMo_Curator_DocumentBatch - The task type consumed and produced by this stage

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment