Implementation:ArroyoSystems Arroyo Compiling State

Field	Value
Sources	ArroyoSystems/arroyo
Domains	Stream_Processing, Distributed_Systems, State_Management
Last Updated	2026-02-08

Overview

Compiling is one of the concrete state implementations in Arroyo's job state machine. It represents the state where a job's compiled program is validated and prepared for scheduling. The Compiling struct implements the State trait, demonstrating the type-safe state machine pattern used throughout the controller's job lifecycle management.

Description

The Compiling state is a pass-through state in the job lifecycle. When a job enters the Compiling state, the compiled program has already been produced during pipeline creation. The state's next method validates that the job should continue running (checking for stop requests or configuration changes), then immediately transitions to the Scheduling state with the program configuration.

This state is representative of the broader State trait pattern used in Arroyo's controller. Each state in the job state machine is implemented as a separate struct in its own module under crates/arroyo-controller/src/states/. All state structs implement the State trait, which requires:

A name() method returning a static string identifier for logging and persistence.
An async next() method that performs the state's work and returns a Transition to the next state or a StateError on failure.

The type-safe transition mechanism works as follows:

Each state struct (e.g., Compiling, Scheduling, Running) is a distinct type.
Valid transitions are declared via TransitionTo trait implementations. For example, Compiling implements TransitionTo<Scheduling>.
The Transition::next(from, to) constructor requires the from type to implement TransitionTo for the to type, making invalid transitions a compile-time error.

Other states in the state machine follow the same pattern with more substantial logic:

Scheduling: Communicates with the scheduler to allocate resources and deploy operators.
Running: Monitors job health, coordinates checkpoints, and handles user commands.
Stopping: Drains operators and releases resources.
Recovering: Restores state from the last checkpoint and restarts the job.

Usage

The Compiling state is entered automatically when a job is first started or when it is recovering/rescaling. It is not invoked directly by user code; rather, the controller's state machine driver transitions jobs into this state as part of the lifecycle.

Code Reference

Source Location

Repository: ArroyoSystems/arroyo (GitHub)
File: crates/arroyo-controller/src/states/compiling.rs
Lines: L1--L17

Signature

#[derive(Debug)]
pub struct Compiling {}

#[async_trait::async_trait]
impl State for Compiling {
    fn name(&self) -> &'static str {
        "Compiling"
    }

    async fn next(self: Box<Self>, ctx: &mut JobContext) -> Result<Transition, StateError> {
        // Pass-through: validates the job should continue,
        // then transitions to Scheduling with the program config
    }
}

Import

use arroyo_controller::states::compiling::Compiling;

I/O Contract

Inputs

Name	Type	Required	Description
`self`	`Box<Compiling>`	Yes	The Compiling state instance (consumed by the transition)
`ctx`	`&mut JobContext`	Yes	Mutable reference to the job context containing the compiled `LogicalProgram`, `JobConfig`, controller handle, and scheduler client

The JobContext provides access to:

program: The compiled LogicalProgram (dataflow graph) to be scheduled.
config: The JobConfig containing parallelism, checkpoint interval, and other execution parameters.
status: The current job status record for checking stop requests or configuration changes.
controller: Handle to the controller for coordinating with other system components.

Outputs

Condition	Type	Description
Normal transition	`Transition::next(Compiling, Scheduling { ... })`	Transitions to the Scheduling state with the program and execution configuration
Stop requested	`Transition::next(Compiling, Stopping { ... })`	If a stop was requested while the job was entering Compiling, transitions to Stopping instead
Error	`StateError`	Returns a state error if the job context is in an inconsistent state, triggering the error recovery path

Usage Examples

State Machine Driver Loop

// The controller's state machine driver runs each job through its states.
// This is illustrative of how the Compiling state is used in context:

let mut current_state: Box<dyn State> = Box::new(Compiling {});

loop {
    let transition = current_state.next(&mut job_context).await;

    match transition {
        Ok(Transition { next_state, .. }) => {
            // Log the transition
            log::info!("Job {} transitioning from {} to {}",
                job_id, current_state.name(), next_state.name());
            current_state = next_state;
        }
        Err(state_error) => {
            // Handle error: may transition to Recovering or Stopped
            current_state = handle_state_error(state_error, &mut job_context);
        }
    }
}

Type-Safe Transition Declaration

// The TransitionTo trait declares valid transitions at compile time.
// These declarations exist in the states module:

impl TransitionTo<Scheduling> for Compiling {}
impl TransitionTo<Stopping> for Compiling {}

// This would NOT compile -- Compiling cannot transition directly to Running:
// impl TransitionTo<Running> for Compiling {}  // Not declared, so:
// Transition::next(Compiling {}, Running {})   // Compile error!

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment