Implementation:ArroyoSystems Arroyo Pipeline Api Types

Knowledge Sources	ArroyoSystems_Arroyo
Domains	Streaming, Pipelines, API
Last Updated	2026-02-08 08:00 GMT

Overview

Defines the API-level types for pipeline CRUD operations, query validation, pipeline graph representation, job status, and log messages used by the Arroyo REST API.

Description

This module provides the full set of request/response types for the pipeline management API:

• PipelinePost -- creation request with name, query, UDFs, parallelism, and optional checkpoint interval.
• PipelinePatch -- partial update for parallelism, checkpoint interval, or stop action.
• PipelineRestart -- restart request with optional force and ignore_state flags.
• Pipeline -- full pipeline representation including id, name, query, UDFs, stop state, graph, and preview flag.
• PreviewPost -- request to run a preview query with optional sink enabling.
• ValidateQueryPost / QueryValidationResult -- query validation request and response with graph and error list.
• PipelineGraph / PipelineNode / PipelineEdge -- the dataflow graph representation with nodes (operators) and edges (streams with key/value/edge types).
• StopType -- enum of stop behaviors: None, Checkpoint, Graceful, Immediate, Force.
• Job -- runtime job status with id, state, run_id, timing, task count, and failure reason.
• JobLogMessage / JobLogLevel -- structured log entries with operator_id, task_index, level, message, details, and error domain.
• OutputData -- preview output data with operator_id, subtask_idx, timestamps, and batch payload.

Usage

Use these types when implementing or consuming the Arroyo REST API for pipeline lifecycle management, query validation, and job monitoring.

Code Reference

Source Location

• Repository: ArroyoSystems_Arroyo
• File: crates/arroyo-rpc/src/api_types/pipelines.rs

Signature

pub struct PipelinePost { pub name: String, pub query: String, pub udfs: Option<Vec<Udf>>, pub parallelism: u64, pub checkpoint_interval_micros: Option<u64> }
pub struct Pipeline { pub id: String, pub name: String, pub query: String, pub udfs: Vec<Udf>, pub graph: PipelineGraph, pub stop: StopType, ... }
pub struct PipelineGraph { pub nodes: Vec<PipelineNode>, pub edges: Vec<PipelineEdge> }
pub struct PipelineNode { pub node_id: u32, pub operator: String, pub description: String, pub parallelism: u32 }
pub struct PipelineEdge { pub src_id: u32, pub dest_id: u32, pub key_type: String, pub value_type: String, pub edge_type: String }
pub enum StopType { None, Checkpoint, Graceful, Immediate, Force }
pub struct Job { pub id: String, pub state: String, pub run_id: u64, pub start_time: Option<u64>, pub failure_reason: Option<FailureReason>, ... }
pub struct JobLogMessage { pub id: String, pub level: JobLogLevel, pub message: String, pub details: String, ... }

Import

use arroyo_rpc::api_types::pipelines::{
    Pipeline, PipelinePost, PipelinePatch, PipelineGraph,
    StopType, Job, JobLogMessage, OutputData,
};

I/O Contract

Inputs

Name	Type	Required	Description
name	String	Yes	Pipeline name
query	String	Yes	SQL query defining the pipeline
parallelism	u64	Yes	Number of parallel tasks per operator
udfs	Option<Vec<Udf>>	No	User-defined functions to include

Outputs

Name	Type	Description
Pipeline	Pipeline	Full pipeline state including compiled graph
Job	Job	Runtime job state with status, timing, and errors
QueryValidationResult	QueryValidationResult	Validation result with graph or error messages

Usage Examples

use arroyo_rpc::api_types::pipelines::{PipelinePost, StopType};

let create_req = PipelinePost {
    name: "my_pipeline".to_string(),
    query: "SELECT * FROM input WHERE value > 100".to_string(),
    udfs: None,
    parallelism: 4,
    checkpoint_interval_micros: Some(10_000_000), // 10 seconds
};

Related Pages

• Environment:ArroyoSystems_Arroyo_Rust_Runtime