Implementation:NVIDIA NeMo Curator NemotronCC Stages

Knowledge Sources	NVIDIA NeMo Curator
Domains	Synthetic Data, Text Transformation, Data Augmentation, NLP
Last Updated	2026-02-14 00:00 GMT

Overview

The Nemotron-CC stages module defines seven concrete processing stages for LLM-based synthetic data generation tasks including Wikipedia paraphrasing, diverse QA generation, text distillation, knowledge extraction, and knowledge list creation, along with associated post-processing stages.

Description

This module contains five BaseSyntheticStage subclasses and two standalone post-processing stages that implement the Nemotron-CC data augmentation pipeline. The Nemotron-CC pipeline transforms web-crawled text (such as Common Crawl data) into higher-quality training data through various LLM-based text transformations.

The five LLM generation stages are:

• WikipediaParaphrasingStage - Rephrases input text in a Wikipedia-like style using the Nemotron-CC system prompt
• DiverseQAStage - Generates diverse question-answer pairs from input text
• DistillStage - Distills input text into a more concise form using a specialized distillation system prompt
• ExtractKnowledgeStage - Extracts structured knowledge statements from input text
• KnowledgeListStage - Generates a bullet-point knowledge list from input text

The two post-processing stages are:

• DiverseQAPostProcessingStage - Parses raw QA output, normalizes bullet formatting, shuffles and samples QA pairs (optionally based on input text token count), and concatenates the selected pairs with the source text
• KnowledgeListPostProcessingStage - Normalizes bullet markers and trims indentation from knowledge list outputs

Usage

Use these stages as components in a Nemotron-CC synthetic data pipeline. The generation stages require an LLM client and model name to be provided (inherited from BaseSyntheticStage). Post-processing stages are placed directly after their corresponding generation stages to clean and format the LLM outputs.

Code Reference

Source Location

• Repository: NeMo-Curator
• File: nemo_curator/stages/synthetic/nemotron_cc/nemotron_cc.py
• Lines: 1-174

Signature

@dataclass
class WikipediaParaphrasingStage(BaseSyntheticStage):
    system_prompt: str = NEMOTRON_CC_SYSTEM_PROMPT
    prompt: str = WIKIPEDIA_REPHRASING_PROMPT_TEMPLATE
    input_field: str = "text"
    output_field: str = "rephrased"

@dataclass
class DiverseQAStage(BaseSyntheticStage):
    system_prompt: str = None
    prompt: str = DIVERSE_QA_PROMPT_TEMPLATE
    input_field: str = "text"
    output_field: str = "diverse_qa"
    tokenizer: AutoTokenizer = None
    prefix: str = "Here are the questions and answers based on the provided text:"
    max_num_pairs: int = 10

@dataclass
class DiverseQAPostProcessingStage(ProcessingStage[DocumentBatch, DocumentBatch]):
    input_field: str = "text"
    qa_field: str = "diverse_qa"
    tokenizer: AutoTokenizer | None = None
    prefix: str = "Here are the questions and answers based on the provided text:"
    max_num_pairs: int = 10
    name: str = "DiverseQAPostProcessing"

@dataclass
class DistillStage(BaseSyntheticStage):
    system_prompt: str = NEMOTRON_CC_DISTILL_SYSTEM_PROMPT
    prompt: str = DISTILL_PROMPT_TEMPLATE
    input_field: str = "text"
    output_field: str = "distill"

@dataclass
class ExtractKnowledgeStage(BaseSyntheticStage):
    system_prompt: str = None
    prompt: str = EXTRACT_KNOWLEDGE_PROMPT_TEMPLATE
    input_field: str = "text"
    output_field: str = "extract_knowledge"

@dataclass
class KnowledgeListStage(BaseSyntheticStage):
    system_prompt: str = None
    prompt: str = KNOWLEDGE_LIST_PROMPT_TEMPLATE
    input_field: str = "text"
    output_field: str = "knowledge_list"

@dataclass
class KnowledgeListPostProcessingStage(ProcessingStage[DocumentBatch, DocumentBatch]):
    input_field: str = "knowledge_list"
    name: str = "KnowledgeListPostProcessing"

Import

from nemo_curator.stages.synthetic.nemotron_cc.nemotron_cc import (
    WikipediaParaphrasingStage,
    DiverseQAStage,
    DiverseQAPostProcessingStage,
    DistillStage,
    ExtractKnowledgeStage,
    KnowledgeListStage,
    KnowledgeListPostProcessingStage,
)

I/O Contract

Inputs (Generation Stages)

Name	Type	Required	Description
input_field	str	Yes	DataFrame column to read source text from (default: "text" for all stages)
client	AsyncLLMClient or LLMClient	Yes	LLM client inherited from BaseSyntheticStage
model_name	str	Yes	LLM model name inherited from BaseSyntheticStage
batch	DocumentBatch	Yes	Input document batch containing source text

Inputs (Post-Processing Stages)

Name	Type	Required	Description
input_field	str	Yes	Column name of the source text (for DiverseQAPostProcessingStage)
qa_field	str	Yes	Column name of the raw QA output to post-process (for DiverseQAPostProcessingStage, default: "diverse_qa")
tokenizer	AutoTokenizer	No	Optional tokenizer for token-count-based QA pair sampling
max_num_pairs	int	No	Maximum number of QA pairs to sample (default: 10)
batch	DocumentBatch	Yes	Input document batch containing raw LLM output

Outputs

Name	Type	Description
result	DocumentBatch	Document batch with the generated/processed text added or updated in the respective output column

Key Implementation Details

DiverseQAPostProcessingStage Processing Logic

The post-processing stage for DiverseQA performs several transformations:

1. Splits the raw LLM output into lines and strips whitespace
2. Removes "- " bullet prefixes from lines
3. Removes the standard prefix line if present
4. Merges question and answer lines into pairs (answers are appended to the preceding question)
5. Shuffles QA pairs randomly
6. Samples a subset of pairs, optionally scaling by input text token count: random.randint(1, max(1, int(max_num_pairs * num_tokens / 150)))
7. Concatenates the original text with the selected QA pairs

# Shuffle the QA pairs and sample up to max_num_pairs
random.shuffle(qa_pairs)
if self.tokenizer is not None:
    num_tokens = len(self.tokenizer.tokenize(text))
    qa_pairs = qa_pairs[: random.randint(1, max(1, int(self.max_num_pairs * num_tokens / 150)))]
else:
    qa_pairs = qa_pairs[: random.randint(1, self.max_num_pairs)]

KnowledgeListPostProcessingStage Formatting

The knowledge list post-processor normalizes bullet formatting by stripping leading "- " or double-space indentation and skipping the first line if it does not start with a bullet:

def _format_text(generated_text: str) -> str:
    lines: list[str] = []
    for idx, line in enumerate(generated_text.split("\n")):
        if idx == 0 and not line.startswith("-"):
            continue
        if line.startswith(("  ", "- ")):
            lines.append(line[2:].strip())
        else:
            lines.append(line)
    return "\n".join(lines)

Stage-Specific Prompt Templates

Each generation stage uses a pre-defined prompt template imported from the prompts module:

Stage	System Prompt	Prompt Template	Output Field
WikipediaParaphrasingStage	NEMOTRON_CC_SYSTEM_PROMPT	WIKIPEDIA_REPHRASING_PROMPT_TEMPLATE	rephrased
DiverseQAStage	None	DIVERSE_QA_PROMPT_TEMPLATE	diverse_qa
DistillStage	NEMOTRON_CC_DISTILL_SYSTEM_PROMPT	DISTILL_PROMPT_TEMPLATE	distill
ExtractKnowledgeStage	None	EXTRACT_KNOWLEDGE_PROMPT_TEMPLATE	extract_knowledge
KnowledgeListStage	None	KNOWLEDGE_LIST_PROMPT_TEMPLATE	knowledge_list

Usage Examples

Wikipedia Paraphrasing

from nemo_curator.stages.synthetic.nemotron_cc.nemotron_cc import WikipediaParaphrasingStage

stage = WikipediaParaphrasingStage(
    client=my_llm_client,
    model_name="nemotron-4-340b",
)

DiverseQA with Post-Processing

from nemo_curator.stages.synthetic.nemotron_cc.nemotron_cc import (
    DiverseQAStage,
    DiverseQAPostProcessingStage,
)
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("nvidia/nemotron-4-340b")

qa_stage = DiverseQAStage(
    client=my_llm_client,
    model_name="nemotron-4-340b",
)

qa_postprocess = DiverseQAPostProcessingStage(
    tokenizer=tokenizer,
    max_num_pairs=8,
)

Knowledge Extraction Pipeline

from nemo_curator.stages.synthetic.nemotron_cc.nemotron_cc import (
    KnowledgeListStage,
    KnowledgeListPostProcessingStage,
)

knowledge_stage = KnowledgeListStage(
    client=my_llm_client,
    model_name="nemotron-4-340b",
)

knowledge_postprocess = KnowledgeListPostProcessingStage()

Related Pages

• Environment:NVIDIA_NeMo_Curator_Python_Linux_Base
• NVIDIA_NeMo_Curator_BaseSyntheticStage - Base class providing LLM interaction logic
• NVIDIA_NeMo_Curator_DataDesignerStage - Alternative synthetic data approach using NeMo Data Designer
• NVIDIA_NeMo_Curator_QAMultilingualSyntheticStage - Related multilingual QA generation stage