Implementation:NVIDIA NeMo Curator Xenna Grafana Dashboard
| Knowledge Sources | |
|---|---|
| Domains | Monitoring, Observability, Grafana |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Pre-configured Grafana dashboard JSON definition for monitoring Ray/Xenna-based data curation workloads, providing over 25 visualization panels for cluster resource utilization, task throughput, actor status, and Ray internals metrics via Prometheus.
Description
The xenna_grafana_dashboard.json file is a standard Grafana dashboard export (3168 lines) that provides out-of-the-box observability for NeMo Curator pipelines running on the Ray/Xenna execution backend. It queries a Prometheus datasource and organizes visualization panels into the following sections:
Default Row:
- Pipeline Progress - Tracks overall pipeline execution progress
- Process Time / Task / Actor - Shows per-task processing latency broken down by actor
- Actor Count - Displays the number of active actors over time
GPU Metrics:
- GPU Allocated vs. Total - Compares allocated GPU resources against total cluster capacity
- GPU Stage Worker Utilization - Shows GPU utilization per worker within each stage
- GPU Utilization per Stage - Aggregated GPU utilization broken down by processing stage
- GPU Utilization - Overall cluster-wide GPU utilization
- GPU Memory per Stage - GPU memory consumption per processing stage
CPU Metrics:
- CPU Allocation per Stage - CPU allocation across processing stages
- CPU Usage per Stage - Actual CPU usage per stage
- CPU Utilization - Overall cluster-wide CPU utilization
- CPU Usage per Actor - Per-actor CPU consumption
Memory and Data Movement:
- System Memory Usage per Stage - System memory consumption per stage
- System Memory Usage per Node - Per-node memory usage
- Cross-Stage Data Movement - Avg Size - Average data transfer size between stages
- Cross-Stage Data Movement - Local Percentage - Percentage of data movement that is node-local
- Comm / Compute Ratio - Communication to computation time ratio
Actor Pool Monitoring:
- Actor Status - Current state of all actors
- Actor Pool - $stage (templated) - Per-stage actor pool status
- Actor Pool Queuing - Queue depth across actor pools
- Actor Pool Queues - $stage (templated) - Per-stage queue details
Ray Internals:
- Ray Spilled - Amount of data spilled from the object store to disk
- Ray Object Store Used - Object store memory usage
- Ray Object Store Available - Remaining object store capacity
- Ray Object Manager Bytes - Object manager memory tracking
The dashboard is titled "Xenna Ray Metrics" (ID: 8), is editable, and uses palette-classic coloring with line-chart visualizations and absolute threshold indicators (green/red at 80%).
Usage
This dashboard JSON is automatically provisioned into Grafana by the write_grafana_configs function when the monitoring stack is started alongside a Xenna executor. Users can also manually import this file into any Grafana instance that has access to the same Prometheus datasource collecting Ray metrics.
Code Reference
Source Location
- Repository: NeMo-Curator
- File: nemo_curator/metrics/xenna_grafana_dashboard.json
- Lines: 1-3168
Key Structure
{
"annotations": { "list": [...] },
"editable": true,
"id": 8,
"panels": [
{ "title": "Default", "type": "row" },
{ "title": "Pipeline Progress", "datasource": { "type": "prometheus" }, ... },
{ "title": "Process Time / Task / Actor", ... },
{ "title": "Actor Count", ... },
{ "title": "GPU Allocated vs. Total", ... },
{ "title": "GPU Utilization per Stage", ... },
{ "title": "CPU Utilization", ... },
{ "title": "System Memory Usage per Stage", ... },
{ "title": "Ray Object Store Used", ... },
...
],
"title": "Xenna Ray Metrics"
}
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| Prometheus datasource | Grafana datasource | Yes | A Prometheus instance (UID: PBFA97CFB590B2093) collecting Ray cluster metrics |
| stage template variable | string | No | Template variable $stage used in per-stage panels for dynamic filtering
|
Outputs
| Name | Type | Description |
|---|---|---|
| Dashboard visualization | Grafana dashboard | Interactive monitoring dashboard with over 25 panels covering GPU, CPU, memory, actor pools, and Ray internals |
Usage Examples
Importing into Grafana
# Import via Grafana API
curl -X POST http://localhost:3000/api/dashboards/db \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GRAFANA_API_KEY" \
-d @nemo_curator/metrics/xenna_grafana_dashboard.json
Automatic Provisioning
# The dashboard is automatically provisioned when starting the monitoring stack
# via the write_grafana_configs utility in the metrics module
from nemo_curator.metrics import write_grafana_configs
write_grafana_configs(output_dir="/etc/grafana/provisioning/dashboards/")