Implementation:NVIDIA NeMo Aligner Process Anthropic HH Chat Prompt

Knowledge Sources	NVIDIA_NeMo_Aligner
Domains	Data Preprocessing, Preference Learning, Constitutional AI
Last Updated	2026-02-08 00:00 GMT

Overview

A script that downloads and processes the Anthropic Helpful-Harmless (HH) dataset, converting it from raw conversation format into chat prompt template format with chosen/rejected preference pairs.

Description

process_anthropic_hh_using_chat_prompt.py processes the Anthropic HH-RLHF dataset from HuggingFace into a format suitable for NeMo Aligner preference training. The script performs the following operations:

Dataset loading: Downloads the Anthropic HH-RLHF dataset from HuggingFace, supporting selective loading of specific subsets (e.g., helpful-base, helpful-online, helpful-rejection-sampled, harmless-base). Multiple subsets are concatenated per split.
Conversation parsing: Parses the raw Anthropic format (with \n\nHuman: and \n\nAssistant: delimiters) into structured user/assistant message pairs.
Chat template formatting: Applies a configurable UserAssistantPromptTemplate to format the conversation pairs using role-specific templates (e.g., <extra_id_*> format). Optionally appends an EOS token.
Preference pair creation: For each sample, the "chosen" and "rejected" responses are formatted independently, and only pairs with matching prompts are retained.
Output generation: Saves two JSONL files per split: one with prompts only and one with chosen/rejected comparisons.

Usage

Use this script when:

You need to prepare the Anthropic HH dataset for preference-based training (DPO, RLHF reward model training)
You want to convert Anthropic conversation format to NeMo chat prompt template format
You need to create helpful-only or full HH dataset subsets

Code Reference

Source Location

Repository: NVIDIA_NeMo_Aligner
File: examples/nlp/cai/process_anthropic_hh_using_chat_prompt.py
Lines: 1-269

Signature

main:

def main():

_process_samples:

def _process_samples(dataset, add_eos: bool, prompt_template_config: dict):

_load_anthropic_dataset:

def _load_anthropic_dataset(dataset_dir_name=None, split_names=None):

_save_dataset:

def _save_dataset(list_of_dict, split: str, save_dir: str, output_file_name_prefix: str, add_eos: bool):

Import

from process_anthropic_hh_using_chat_prompt import _process_samples, _load_anthropic_dataset

I/O Contract

Inputs

Name	Type	Required	Description
--output-dir	`str`	Yes	Directory to write output files
--dataset-dir-name	`str` (multiple)	No	Anthropic HH subsets to include (e.g., helpful-base, helpful-online, helpful-rejection-sampled, harmless-base). If omitted, loads the full HH-RLHF dataset.
--output-file-name-prefix	`str`	No	Prefix for output file names (default: "anthropic_hh" when using full dataset)
--add-eos	`str`	No	Whether to append EOS token to comparisons (default: "True")
--user_format	`str`	No	User message format template (default: "<extra_id_1>User\n{MESSAGE}\n<extra_id_1>Assistant\n")
--assistant_format	`str`	No	Assistant message format template (default: "{MESSAGE}\n")
--system_format	`str`	No	System message format template (default: "<extra_id_0>System\n{MESSAGE}\n")
--system_default_message	`str`	No	Default system message (default: "")
--bos_token	`str`	No	Begin-of-sequence token (default: None)
--eos_token	`str`	No	End-of-sequence token (default: "<extra_id_1>")
--response_extract_pattern	`str`	No	Pattern for extracting responses (default: "<extra_id_1>Assistant\n")

Outputs

Name	Type	Description
{prefix}_{split}_prompts_with_chat_prompt.jsonl	JSONL file	Prompts formatted with chat template, one per line as {"text": "..."}
{prefix}_{split}_comparisons_with_chat_prompt.jsonl	JSONL file	Chosen/rejected pairs formatted with chat template, alternating lines as {"text": "..."}

Usage Examples

# Preprocess Anthropic helpful-only dataset:
python process_anthropic_hh_using_chat_prompt.py \
    --output-dir /data/processed \
    --dataset-dir-name helpful-base helpful-online helpful-rejection-sampled \
    --output-file-name-prefix anthropic_helpful_only

# Preprocess full Anthropic HH dataset:
python process_anthropic_hh_using_chat_prompt.py \
    --output-dir /data/processed \
    --output-file-name-prefix anthropic_hh_full

# Preprocess with custom prompt template (Mistral format):
python process_anthropic_hh_using_chat_prompt.py \
    --output-dir /data/processed \
    --user_format "[INST] {MESSAGE} [/INST]" \
    --assistant_format "{MESSAGE}</s> " \
    --bos_token "<s>" \
    --eos_token "</s>" \
    --response_extract_pattern "[/INST]"

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment