Implementation:ArroyoSystems Arroyo Delta Connector

Knowledge Sources	ArroyoSystems_Arroyo
Domains	Streaming, Connectors
Last Updated	2026-02-08 08:00 GMT

Overview

DeltaLakeConnector implements the Arroyo Connector trait for writing streaming data to Delta Lake tables on object storage with optional partition-based shuffling.

Description

The Delta Lake connector is a sink-only connector (hidden from the UI) that writes data to Delta Lake tables at a specified object storage path. It supports configurable partitioning via PartitioningConfig and partition-based data shuffling when shuffle_by_partition is enabled. The connector validates the storage path using BackendConfig::parse_url and delegates the actual writing to the filesystem sink infrastructure via make_sink with TableFormat::Delta. It requires both a schema and a format definition. The DeltaLakeTable configuration includes DeltaLakeSink with fields for path, storage options, rolling policy, file naming, partitioning, multipart upload settings, and version.

Usage

Use DeltaLakeConnector when you need to write Arroyo pipeline results to Delta Lake tables stored on S3, GCS, or other object storage backends.

Code Reference

Source Location

• Repository: ArroyoSystems_Arroyo
• File: crates/arroyo-connectors/src/filesystem/delta.rs

Signature

pub struct DeltaLakeConnector {}

impl Connector for DeltaLakeConnector {
    type ProfileT = EmptyConfig;
    type TableT = DeltaLakeTable;

    fn name(&self) -> &'static str; // returns "delta"
    fn metadata(&self) -> arroyo_rpc::api_types::connections::Connector;
    fn table_type(&self, _: EmptyConfig, _: DeltaLakeTable) -> ConnectionType; // always Sink
    fn from_config(&self, id: Option<i64>, name: &str, config: EmptyConfig,
        table: DeltaLakeTable, schema: Option<&ConnectionSchema>) -> anyhow::Result<Connection>;
    fn make_operator(&self, _: EmptyConfig, table: DeltaLakeTable,
        config: OperatorConfig) -> anyhow::Result<ConstructedOperator>;
}

Import

use arroyo_connectors::filesystem::delta::DeltaLakeConnector;

I/O Contract

Inputs

Name	Type	Required	Description
path	String	Yes	Object storage path for the Delta Lake table (e.g. s3://bucket/table)
format	Format	Yes	Serialization format for the data files
schema	ConnectionSchema	Yes	Schema for the Delta Lake table
partitioning	PartitioningConfig	No	Time-based or expression-based partitioning configuration
rolling_policy	RollingPolicy	No	File rolling policy for output files

Outputs

Name	Type	Description
Delta Lake files	Parquet files	Data files written to the Delta Lake table location with Delta transaction log

Usage Examples

CREATE TABLE delta_sink (
    user_id TEXT,
    event TEXT
) WITH (
    connector = 'delta',
    path = 's3://my-bucket/events',
    format = 'parquet'
);

Related Pages

• Environment:ArroyoSystems_Arroyo_Rust_Runtime