Implementation:NVIDIA NeMo Curator Nightly Benchmark Config
| Knowledge Sources | |
|---|---|
| Domains | Benchmarking, Configuration, CI/CD |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
YAML configuration file that defines the complete nightly benchmark suite for NeMo Curator, specifying datasets, reporting sinks, Ray resource allocations, and approximately 20 benchmark entries covering all major pipeline types and executor backends.
Description
The nightly-benchmark.yaml file serves as the production nightly regression configuration for the NeMo Curator benchmarking framework. It is structured into several top-level sections:
Path Configuration: Three base paths (results_path, datasets_path, model_weights_path) are defined at the top of the file and are used as template placeholders throughout the configuration. These are resolved at runtime via the {datasets_path}, {model_weights_path}, and {session_entry_dir} placeholder syntax.
Datasets: Twelve named datasets are declared, each with one or more format variants (parquet, jsonl, wds, tar, mp4, bin, json, files). These include tinystories (parquet and jsonl), commoncrawl (jsonl and parquet with Gemma embeddings), mscoco (WebDataset), videos (mp4), rpv2-2023-14-en (parquet), arxiv_downloads (tar), fasttext_model (bin), and gretel_symptoms (jsonl).
Sinks: Optional reporting sinks are configured, with Slack enabled by default (using the ${SLACK_WEBHOOK_URL} environment variable). MLflow and Google Drive sinks are present but commented out.
Global Settings: A default timeout of 7200 seconds (2 hours), a 500 GB Ray object store size, and delete_scratch: true for cleanup after each entry.
Benchmark Entries: Each entry in the entries list specifies a benchmark scenario with:
- name: A unique identifier (e.g.,
domain_classification_raydata) - script: The Python benchmark script to execute
- args: Command-line arguments with template substitution for paths and datasets
- timeout_s: Per-entry timeout override
- ray: Resource requirements (num_cpus, num_gpus, enable_object_spilling)
- sink_data: Additional metrics to report to each sink
- requirements: Pass/fail criteria using exact_value, min_value, and max_value thresholds
The benchmark entries cover the following categories:
- Domain classification (ray_data and xenna executors)
- Embedding generation (ray_data and xenna executors)
- Exact dedup identification
- Fuzzy dedup identification
- Semantic dedup identification (xenna executor)
- Dedup removal (ray_data and xenna executors)
- Score filtering (ray_data and xenna executors)
- Modifier benchmarks (ray_data and xenna executors)
- Image curation pipeline
- Audio FLEURS benchmark
- ArXiv end-to-end pipeline (ray_data and xenna executors)
- Video embedding, transcoding, and captioning
- NDD via NVIDIA NIM
- Video TransNetV2 motion/aesthetic filtering with embeddings
Usage
This configuration file is consumed by the benchmark runner (benchmarking/run.py) via the --config command-line argument. It is intended for nightly CI/CD runs that validate both performance and correctness across all modalities (text, image, audio, video) and executor backends (ray_data, xenna).
Code Reference
Source Location
- Repository: NeMo-Curator
- File: benchmarking/nightly-benchmark.yaml
- Lines: 1-664
Key Structure
results_path: /path/where/results/are/stored
datasets_path: /path/to/datasets
model_weights_path: /path/to/model_weights
datasets:
- name: "tinystories"
formats:
- type: "parquet"
path: "{datasets_path}/tinystories/parquet_data"
default_timeout_s: 7200
object_store_size: 536870912000 # 500GB
delete_scratch: true
sinks:
- name: slack
enabled: true
webhook_url: ${SLACK_WEBHOOK_URL}
default_metrics: ["exec_time_s"]
entries:
- name: domain_classification_raydata
enabled: true
script: domain_classification_benchmark.py
args: >-
--benchmark-results-path={session_entry_dir}
--executor=ray_data
--input-path={dataset:tinystories,parquet}
timeout_s: 1000
ray:
num_cpus: 64
num_gpus: 4
requirements:
- metric: throughput_docs_per_sec
min_value: 2565
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| results_path | string | Yes | Base directory where benchmark session results are stored |
| datasets_path | string | Yes | Base directory containing all benchmark datasets |
| model_weights_path | string | Yes | Base directory containing model weight files |
| datasets | list | Yes | Named dataset definitions with format and path specifications |
| sinks | list | No | Optional reporting integrations (Slack, MLflow, Google Drive) |
| default_timeout_s | int | No | Default timeout in seconds for benchmark entries (default: 7200) |
| object_store_size | int/float/string | No | Ray object store size in bytes, as a fraction, or "default" |
| delete_scratch | bool | No | Whether to delete scratch directories after each entry run |
| entries | list | Yes | List of benchmark entry definitions to execute |
Outputs
| Name | Type | Description |
|---|---|---|
| Session directory | directory | Contains per-entry subdirectories with results.json, params.json, metrics.json, tasks.pkl, and logs |
| Sink reports | varies | Slack messages (or MLflow/GDrive artifacts) containing execution metrics for each entry |
Usage Examples
Running the Nightly Benchmark Suite
# Run all enabled entries from the nightly config
python benchmarking/run.py --config benchmarking/nightly-benchmark.yaml
# Run with a custom session name
python benchmarking/run.py --config benchmarking/nightly-benchmark.yaml \
--session-name "nightly-2026-02-14"
# Filter to only dedup-related entries
python benchmarking/run.py --config benchmarking/nightly-benchmark.yaml \
--entries "dedup and not removal"
# List entries without running
python benchmarking/run.py --config benchmarking/nightly-benchmark.yaml --list
Benchmark Entry with Requirements
# Example entry with exact and range-based requirements
- name: fuzzy_dedup_identification
enabled: true
script: fuzzy_dedup_identification_benchmark.py
args: >-
--benchmark-results-path={session_entry_dir}
--input-path={dataset:commoncrawl,jsonl}
--cache-path={session_entry_dir}/scratch/cache
--output-path={session_entry_dir}/scratch/output
--bands-per-iteration=20
timeout_s: 700
ray:
num_cpus: 64
num_gpus: 4
enable_object_spilling: false
requirements:
- metric: num_duplicates
min_value: 58633871
max_value: 59818394