Implementation:NVIDIA NeMo Curator Slicer

Knowledge Sources	NVIDIA NeMo Curator
Domains	Data_Curation, Text_Processing, Text_Extraction
Last Updated	2026-02-14 00:00 GMT

Overview

Slicer is a document modifier that extracts a substring from a document based on integer indices or string markers, with configurable inclusion or exclusion of boundary markers and optional stripping.

Description

Slicer extends DocumentModifier and provides a flexible text extraction mechanism. It accepts left and right boundary parameters that can each be either an integer index or a string marker:

Integer boundaries:

• left as int: Slice starts at this character index (inclusive).
• right as int: Slice ends at this character index (exclusive, standard Python slicing).

String boundaries:

• left as str: Finds the first occurrence of this substring using str.find(). If include_left is True, the slice starts at the beginning of the match. If False, the slice starts after the match.
• right as str: Finds the last occurrence of this substring using str.rfind(). If include_right is True, the slice ends after the match. If False, the slice ends at the beginning of the match.

If a string marker is not found in the text, the modifier returns an empty string. The strip parameter (default True) controls whether the resulting substring is stripped of leading and trailing whitespace.

Usage

Use Slicer when documents have a known structure with identifiable headers, footers, or section markers that you want to extract or remove. For example, extracting the main body of an article by slicing between the title and the footer, or removing everything before a known starting marker.

Code Reference

Source Location

• Repository: NeMo-Curator
• File: nemo_curator/stages/text/modifiers/slicer.py
• Lines: 1-78

Signature

class Slicer(DocumentModifier):
    def __init__(
        self,
        left: int | str | None = 0,
        right: int | str | None = None,
        include_left: bool = True,
        include_right: bool = True,
        strip: bool = True,
    ):
        ...

    def modify_document(self, text: str) -> str:
        ...

Import

from nemo_curator.stages.text.modifiers.slicer import Slicer

I/O Contract

Inputs

Name	Type	Required	Description
left	str | None	No (default: 0)	Left boundary of the slice. Integer for character index, string for marker-based search (first occurrence).
right	str | None	No (default: None)	Right boundary of the slice. Integer for character index, string for marker-based search (last occurrence). None means end of text.
include_left	bool	No (default: True)	When left is a string, whether to include the marker text itself in the result.
include_right	bool	No (default: True)	When right is a string, whether to include the marker text itself in the result.
strip	bool	No (default: True)	Whether to strip leading/trailing whitespace from the result.
text	str	Yes	The document text passed to modify_document().

Outputs

Name	Type	Description
return value	str	The extracted substring. Returns an empty string if a string marker is not found in the text.

Usage Examples

Integer Index Slicing

from nemo_curator.stages.text.modifiers.slicer import Slicer

# Extract characters 10 through 50
modifier = Slicer(left=10, right=50)

text = "HEADER    This is the main content of the document that continues..."
result = modifier.modify_document(text)
# Returns characters at indices 10-49, stripped

String Marker Slicing

from nemo_curator.stages.text.modifiers.slicer import Slicer

# Extract text between markers, excluding the markers themselves
modifier = Slicer(
    left="BEGIN_CONTENT",
    right="END_CONTENT",
    include_left=False,
    include_right=False,
)

text = "Header stuff BEGIN_CONTENT Main article body END_CONTENT Footer stuff"
result = modifier.modify_document(text)
# Returns "Main article body"

Including Markers

from nemo_curator.stages.text.modifiers.slicer import Slicer

# Extract text from a marker to the end, including the marker
modifier = Slicer(left="Chapter 1", right=None, include_left=True)

text = "Table of Contents\nChapter 1\nThe story begins here..."
result = modifier.modify_document(text)
# Returns "Chapter 1\nThe story begins here..."

Missing Marker Returns Empty String

from nemo_curator.stages.text.modifiers.slicer import Slicer

modifier = Slicer(left="MISSING_MARKER", right=None)

text = "This text does not contain the marker."
result = modifier.modify_document(text)
# Returns "" (empty string) because the left marker was not found

Related Pages

• Implementation:NVIDIA_NeMo_Curator_DocumentModifier — Base class that this modifier extends
• Implementation:NVIDIA_NeMo_Curator_LineRemover — Alternative modifier for line-level removal
• Environment:NVIDIA_NeMo_Curator_Python_Linux_Base