Principle:ArroyoSystems Arroyo Job Monitoring
Metadata
| Field | Value |
|---|---|
| Page Type | Principle |
| Knowledge Sources | Repo (ArroyoSystems/arroyo), Doc (Arroyo Documentation) |
| Domains | Stream_Processing, Observability |
| Last Updated | 2026-02-08 |
Overview
Collecting and exposing operator-level runtime metrics for running streaming jobs, providing throughput and backpressure visibility through a REST API endpoint that bridges to the controller's gRPC metrics service.
Description
Job monitoring provides real-time observability into streaming pipeline performance at the operator and subtask level. The monitoring system aggregates metrics from distributed worker processes and exposes them through a unified API endpoint.
Metric Collection Architecture
Each worker process maintains per-operator, per-subtask Prometheus counters tracking messages sent/received, bytes sent/received, batches processed, and deserialization errors. These counters are labeled with operator ID, subtask index, and operator name. The controller aggregates metrics from all workers via gRPC.
API Bridge Pattern
The metrics REST endpoint acts as a bridge between HTTP clients and the gRPC-based controller service. It establishes a gRPC connection with zstd compression, sends a metrics request for a specific job, and deserializes the JSON response into structured operator metric groups. When a job is not running, the endpoint returns an empty collection rather than an error, providing graceful degradation.
Operator Metric Groups
Metrics are organized hierarchically: each operator has a group containing subtask-level metrics. This structure maps directly to the dataflow graph topology and parallelism configuration, allowing the web UI to render per-operator throughput charts and identify bottleneck operators through backpressure measurements.
Usage
Job monitoring is applied in the following scenarios:
- Pipeline dashboard: The web UI periodically polls operator metrics to display real-time throughput and backpressure charts on the pipeline detail page.
- Performance tuning: Engineers identify slow operators or backpressure hotspots to guide parallelism adjustments.
- Alerting integration: Monitoring systems can poll the metrics endpoint to trigger alerts on throughput degradation.