Implementation:NVIDIA NeMo Curator QuotationRemover

Knowledge Sources	NVIDIA NeMo Curator
Domains	Data_Curation, Text_Cleaning
Last Updated	2026-02-14 00:00 GMT

Overview

QuotationRemover is a document modifier that removes wrapping double quotation marks from documents that appear to be entirely quoted, while preserving legitimate quotation structures.

Description

QuotationRemover extends DocumentModifier and implements a heuristic for detecting and removing spurious wrapping quotes from document text. The logic in modify_document() applies the following rules:

1. If the stripped text is 2 characters or fewer, it is returned unchanged (too short to meaningfully unquote).
2. If the text starts with " and ends with ":
   • If the text contains no newlines, the outer quotes are removed.
   • If the text contains newlines, the outer quotes are removed only if the first line does not end with a quotation mark. This check prevents removal of wrapping quotes when the text contains legitimately quoted multi-paragraph content (where the first line ending with " suggests the opening quote is part of a meaningful quotation structure).

This heuristic targets a common artifact of web scraping and text extraction, where entire document bodies may be incorrectly wrapped in quotation marks.

Usage

Use QuotationRemover when processing web-scraped text that may contain spurious wrapping quotation marks as an artifact of the extraction process. It is typically used as part of a text cleaning pipeline via the Modify processing stage.

Code Reference

Source Location

• Repository: NeMo-Curator
• File: nemo_curator/stages/text/modifiers/quotation_remover.py
• Lines: 1-37

Signature

class QuotationRemover(DocumentModifier):
    def __init__(self):
        ...

    def modify_document(self, text: str) -> str:
        ...

Import

from nemo_curator.stages.text.modifiers.quotation_remover import QuotationRemover

I/O Contract

Inputs

Name	Type	Required	Description
text	str	Yes	The document text passed to modify_document().

Outputs

Name	Type	Description
return value	str	The document text with wrapping quotation marks removed if the heuristic conditions are met, or the original text unchanged otherwise.

Decision Logic

The following table summarizes the removal behavior:

Condition	Result
Text length <= 2 (stripped)	Returned unchanged
Does not start and end with "	Returned unchanged
Starts/ends with ", no newlines	Outer quotes removed
Starts/ends with ", has newlines, first line does NOT end with "	Outer quotes removed
Starts/ends with ", has newlines, first line ends with "	Returned unchanged (legitimate quotation structure)

Usage Examples

Basic Usage

from nemo_curator.stages.text.modifiers.quotation_remover import QuotationRemover

modifier = QuotationRemover()

# Single-line wrapped text
text = '"This is a document that was incorrectly wrapped in quotes."'
result = modifier.modify_document(text)
# Returns 'This is a document that was incorrectly wrapped in quotes.'

Multi-Line Text

from nemo_curator.stages.text.modifiers.quotation_remover import QuotationRemover

modifier = QuotationRemover()

# Multi-line text with wrapping quotes (first line does not end with quote)
text = '"This is the first line\nThis is the second line"'
result = modifier.modify_document(text)
# Returns 'This is the first line\nThis is the second line'

Preserved Legitimate Quotes

from nemo_curator.stages.text.modifiers.quotation_remover import QuotationRemover

modifier = QuotationRemover()

# Multi-line text where first line ends with quote (legitimate structure)
text = '"He said hello"\n"She said goodbye"'
result = modifier.modify_document(text)
# Returns the text unchanged because the first line ends with a quote

Related Pages

• NVIDIA_NeMo_Curator_DocumentModifier — Base class that this modifier extends
• Environment:NVIDIA_NeMo_Curator_Python_Linux_Base