Jump to content

Connect SuperML | Leeroopedia MCP: Equip your AI agents with best practices, code verification, and debugging knowledge. Powered by Leeroo — building Organizational Superintelligence. Contact us at founders@leeroo.com.

Implementation:NVIDIA NeMo Curator Xenna Grafana Dashboard

From Leeroopedia
Knowledge Sources
Domains Monitoring, Observability, Grafana
Last Updated 2026-02-14 00:00 GMT

Overview

Pre-configured Grafana dashboard JSON definition for monitoring Ray/Xenna-based data curation workloads, providing over 25 visualization panels for cluster resource utilization, task throughput, actor status, and Ray internals metrics via Prometheus.

Description

The xenna_grafana_dashboard.json file is a standard Grafana dashboard export (3168 lines) that provides out-of-the-box observability for NeMo Curator pipelines running on the Ray/Xenna execution backend. It queries a Prometheus datasource and organizes visualization panels into the following sections:

Default Row:

  • Pipeline Progress - Tracks overall pipeline execution progress
  • Process Time / Task / Actor - Shows per-task processing latency broken down by actor
  • Actor Count - Displays the number of active actors over time

GPU Metrics:

  • GPU Allocated vs. Total - Compares allocated GPU resources against total cluster capacity
  • GPU Stage Worker Utilization - Shows GPU utilization per worker within each stage
  • GPU Utilization per Stage - Aggregated GPU utilization broken down by processing stage
  • GPU Utilization - Overall cluster-wide GPU utilization
  • GPU Memory per Stage - GPU memory consumption per processing stage

CPU Metrics:

  • CPU Allocation per Stage - CPU allocation across processing stages
  • CPU Usage per Stage - Actual CPU usage per stage
  • CPU Utilization - Overall cluster-wide CPU utilization
  • CPU Usage per Actor - Per-actor CPU consumption

Memory and Data Movement:

  • System Memory Usage per Stage - System memory consumption per stage
  • System Memory Usage per Node - Per-node memory usage
  • Cross-Stage Data Movement - Avg Size - Average data transfer size between stages
  • Cross-Stage Data Movement - Local Percentage - Percentage of data movement that is node-local
  • Comm / Compute Ratio - Communication to computation time ratio

Actor Pool Monitoring:

  • Actor Status - Current state of all actors
  • Actor Pool - $stage (templated) - Per-stage actor pool status
  • Actor Pool Queuing - Queue depth across actor pools
  • Actor Pool Queues - $stage (templated) - Per-stage queue details

Ray Internals:

  • Ray Spilled - Amount of data spilled from the object store to disk
  • Ray Object Store Used - Object store memory usage
  • Ray Object Store Available - Remaining object store capacity
  • Ray Object Manager Bytes - Object manager memory tracking

The dashboard is titled "Xenna Ray Metrics" (ID: 8), is editable, and uses palette-classic coloring with line-chart visualizations and absolute threshold indicators (green/red at 80%).

Usage

This dashboard JSON is automatically provisioned into Grafana by the write_grafana_configs function when the monitoring stack is started alongside a Xenna executor. Users can also manually import this file into any Grafana instance that has access to the same Prometheus datasource collecting Ray metrics.

Code Reference

Source Location

  • Repository: NeMo-Curator
  • File: nemo_curator/metrics/xenna_grafana_dashboard.json
  • Lines: 1-3168

Key Structure

{
    "annotations": { "list": [...] },
    "editable": true,
    "id": 8,
    "panels": [
        { "title": "Default", "type": "row" },
        { "title": "Pipeline Progress", "datasource": { "type": "prometheus" }, ... },
        { "title": "Process Time / Task / Actor", ... },
        { "title": "Actor Count", ... },
        { "title": "GPU Allocated vs. Total", ... },
        { "title": "GPU Utilization per Stage", ... },
        { "title": "CPU Utilization", ... },
        { "title": "System Memory Usage per Stage", ... },
        { "title": "Ray Object Store Used", ... },
        ...
    ],
    "title": "Xenna Ray Metrics"
}

I/O Contract

Inputs

Name Type Required Description
Prometheus datasource Grafana datasource Yes A Prometheus instance (UID: PBFA97CFB590B2093) collecting Ray cluster metrics
stage template variable string No Template variable $stage used in per-stage panels for dynamic filtering

Outputs

Name Type Description
Dashboard visualization Grafana dashboard Interactive monitoring dashboard with over 25 panels covering GPU, CPU, memory, actor pools, and Ray internals

Usage Examples

Importing into Grafana

# Import via Grafana API
curl -X POST http://localhost:3000/api/dashboards/db \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -d @nemo_curator/metrics/xenna_grafana_dashboard.json

Automatic Provisioning

# The dashboard is automatically provisioned when starting the monitoring stack
# via the write_grafana_configs utility in the metrics module
from nemo_curator.metrics import write_grafana_configs

write_grafana_configs(output_dir="/etc/grafana/provisioning/dashboards/")

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment