Implementation:Huggingface Datatrove TextUtils
| Knowledge Sources | |
|---|---|
| Domains | Text Processing, NLP |
| Last Updated | 2026-02-14 17:00 GMT |
Overview
Provides text normalization, n-gram generation, and text splitting utilities used throughout the datatrove processing and deduplication pipeline.
Description
The TextUtils module is a core utility library that supplies configurable text normalization, n-gram extraction, and multi-mode text splitting functions. The primary entry point is the simplify_text function, which applies a sequence of normalization steps controlled by a TextNormConfig dataclass: lowercasing, number normalization (using a Unicode-aware regex pattern that matches digits in any script), punctuation removal via a translation table, whitespace collapsing, and Unicode diacritics stripping through NFD decomposition.
The module defines two extensive character sets: PUNCTUATION, a broad collection of punctuation characters from many scripts including control characters, and TERMINAL_PUNCTUATION, a set of sentence-ending punctuation marks from a wide range of writing systems. These sets are used to build a translation table (PUNCTUATION_TRANS) that converts all punctuation to spaces.
The ngrams function generates n-gram sequences from any iterable using itertools.tee for memory-efficient sliding window iteration. The split_into_parts function provides a unified, cached interface for splitting text into documents, sentences, words, or paragraphs, delegating to language-specific word tokenizers for sentence and word modes. Convenience wrappers split_into_words, split_into_sentences, and split_into_paragraphs simplify common use cases.
Usage
Use these utilities when normalizing text for comparison (e.g., deduplication, decontamination), generating n-gram shingles for hashing, or splitting text into structural units (sentences, words, paragraphs) for quality analysis or filtering.
Code Reference
Source Location
- Repository: Huggingface_Datatrove
- File: src/datatrove/utils/text.py
- Lines: 1-317
Signature
@dataclass
class TextNormConfig:
lowercase: bool = True
norm_whitespace: bool = True
remove_punctuation: bool = True
norm_unicode_diacritics: bool = True
norm_numbers: bool = True
norm_weekdays: bool = False
norm_monthnames: bool = False
def simplify_text(text: str, config=DEF_TEXT_NORM_CONFIG) -> str: ...
def ngrams(sequence: Iterable, n: int): ...
def split_into_parts(text, mode="DOCUMENT", language=Languages.english): ...
def split_into_words(text, language=Languages.english): ...
def split_into_sentences(text, language=Languages.english): ...
def split_into_paragraphs(text, language=Languages.english): ...
Import
from datatrove.utils.text import simplify_text, TextNormConfig, ngrams, split_into_parts
from datatrove.utils.text import split_into_words, split_into_sentences, split_into_paragraphs
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| text | str | Yes | The input text string to normalize or split |
| config | TextNormConfig | No | Configuration controlling which normalization steps to apply (defaults to all enabled except weekday/month normalization) |
| mode | str | No | Splitting mode: "DOCUMENT", "SENTENCE", "WORDS", or "PARAGRAPH" |
| language | str | No | Language code for tokenizer selection (defaults to English) |
| sequence | Iterable | Yes (ngrams) | Any iterable to generate n-grams from |
| n | int | Yes (ngrams) | The n-gram size |
Outputs
| Name | Type | Description |
|---|---|---|
| simplified text | str | The normalized text string (from simplify_text) |
| n-grams | iterator of tuples | Sliding window tuples of size n (from ngrams) |
| parts | list[str] | Text split into the requested units (from split_into_parts) |
Usage Examples
Basic Usage
from datatrove.utils.text import simplify_text, TextNormConfig, ngrams
# Normalize text with default config
normalized = simplify_text("Hello, World! 123 numbers.")
# Result: "hello world 0 numbers"
# Custom config: keep punctuation
config = TextNormConfig(remove_punctuation=False)
normalized = simplify_text("Hello, World!", config)
# Generate bigrams
tokens = ["the", "quick", "brown", "fox"]
bigrams = list(ngrams(tokens, 2))
# Result: [("the", "quick"), ("quick", "brown"), ("brown", "fox")]
# Split text into sentences
from datatrove.utils.text import split_into_sentences
sentences = split_into_sentences("Hello world. How are you?")