Implementation:NVIDIA NeMo Curator Text Utils

Knowledge Sources	NVIDIA NeMo Curator
Domains	Data Curation, Text Processing, NLP Utilities
Last Updated	2026-02-14 00:00 GMT

Overview

The text_utils module provides a collection of text processing utility functions for tokenization, segmentation, n-gram extraction, code analysis, and boilerplate detection used across the NeMo Curator pipeline.

Description

This module contains several categories of utility functions:

Word Splitting: The get_word_splitter(language) function returns a language-specific word splitter based on ISO 639-1 language codes. For Chinese ("zh"), it uses the Jieba library for word segmentation. For Japanese ("ja"), it uses MeCab morphological analysis. For all other languages, it defaults to whitespace splitting.

Document Segmentation: get_paragraphs(document) splits text by double newlines ("\n\n"), while get_sentences(document) splits by single newlines, filtering out empty lines.

N-gram Extraction: get_ngrams(input_list, n) efficiently generates n-grams from a token list using zip offsets.

Boilerplate Detection: is_paragraph_indices_in_top_or_bottom_only determines whether a set of paragraph indices are contiguously located only at the top or bottom of a document, which is used to identify boilerplate content such as headers and footers.

Code Analysis: A set of functions (get_comments_and_docstring, get_comments, get_docstrings, parse_docstrings) use Python's ast and tokenize modules to extract docstrings and comments from Python source code. The NODE_TYPES dictionary maps AST node types (ClassDef, FunctionDef, Module) to human-readable labels.

Word Extraction: get_words(text) provides lowercase, punctuation-stripped word extraction with character position tracking, returning both the word list and the starting character positions of each word.

Usage

These utilities are used internally by text quality heuristic filters and classifiers to analyze document structure, compute linguistic features, and detect boilerplate content. They support multiple languages and are essential building blocks for the text curation pipeline.

Code Reference

Source Location

• Repository: NeMo-Curator
• File: nemo_curator/stages/text/utils/text_utils.py
• Lines: 1-218

Key Functions

def get_word_splitter(language: str) -> Callable[[str], list[str]]: ...
def get_paragraphs(document: str) -> list[str]: ...
def get_sentences(document: str) -> list[str]: ...
def get_ngrams(input_list: list[str], n: int) -> list[tuple[str, ...]]: ...
def is_paragraph_indices_in_top_or_bottom_only(
    boilerplate_paragraph_indices: list[int],
    num_paragraphs: int,
) -> bool: ...
def get_comments_and_docstring(
    source: str, comments: bool = True, clean_comments: bool = False,
) -> tuple[str, str]: ...
def get_comments(s: str, clean: bool = False) -> str: ...
def get_docstrings(source: str, module: str = "<string>") -> list[str]: ...
def parse_docstrings(source: str) -> list[tuple[ast.AST, str | None, str]]: ...
def remove_punctuation(str_in: str) -> str: ...
def get_words(text: str) -> tuple[list[str], list[int]]: ...

Import

from nemo_curator.stages.text.utils.text_utils import (
    get_word_splitter,
    get_paragraphs,
    get_sentences,
    get_ngrams,
    get_words,
    get_comments_and_docstring,
)

I/O Contract

get_word_splitter

Name	Type	Required	Description
language	str	Yes	ISO 639-1 language code (e.g., "en", "zh", "ja")

Returns a callable Callable[[str], list[str]] that splits text into words.

get_ngrams

Name	Type	Required	Description
input_list	list[str]	Yes	List of tokens to generate n-grams from
n	int	Yes	Size of the n-grams

Returns list[tuple[str, ...]] of n-gram tuples.

get_words

Name	Type	Required	Description
text	str	Yes	Input text to extract words from

Returns tuple[list[str], list[int]] containing lowercase words and their starting character positions.

is_paragraph_indices_in_top_or_bottom_only

Name	Type	Required	Description
boilerplate_paragraph_indices	list[int]	Yes	Sorted list of paragraph indices identified as boilerplate
num_paragraphs	int	Yes	Total number of paragraphs in the document

Returns bool indicating whether the indices are contiguously at the top, bottom, or both.

Usage Examples

Language-Aware Word Splitting

from nemo_curator.stages.text.utils.text_utils import get_word_splitter

# English: whitespace splitting
en_splitter = get_word_splitter("en")
words = en_splitter("Hello world")  # ["Hello", "world"]

# Chinese: Jieba segmentation
zh_splitter = get_word_splitter("zh")
words = zh_splitter("你好世界")  # Jieba word segments

N-gram Extraction

from nemo_curator.stages.text.utils.text_utils import get_ngrams

tokens = ["the", "quick", "brown", "fox"]
bigrams = get_ngrams(tokens, 2)
# [("the", "quick"), ("quick", "brown"), ("brown", "fox")]

Extracting Code Documentation

from nemo_curator.stages.text.utils.text_utils import get_comments_and_docstring

source_code = '''
def hello():
    """Say hello."""
    # Print greeting
    print("hello")
'''
docstrings, comments = get_comments_and_docstring(source_code)

Related Pages

• Environment:NVIDIA_NeMo_Curator_Python_Linux_Base
• NVIDIA_NeMo_Curator_DocumentSplitter - Uses text segmentation concepts from this module
• NVIDIA_NeMo_Curator_ScoreFilter - Heuristic filters that use these text utility functions