Implementation:ArroyoSystems Arroyo Redis Lookup

Knowledge Sources	ArroyoSystems_Arroyo
Domains	Streaming, Connectors
Last Updated	2026-02-08 08:00 GMT

Overview

RedisLookup implements the Arroyo LookupConnector trait to perform key-based lookups against Redis using MGET commands for lookup table joins.

Description

The RedisLookup connector performs batch key lookups against Redis using the MGET command for efficient multi-key retrieval. It requires exactly one key array of DataType::Utf8 type. The lookup method lazily initializes a Redis connection on first use. Each returned Redis value is deserialized via an ArrowDeserializer: Nil values produce null rows, SimpleString and BulkString values are deserialized from their byte representation, and any other Redis value type causes a panic. The connector supports metadata fields, currently exposing the key field which maps the lookup key into the output schema. Each deserialized record includes a LOOKUP_KEY_INDEX_FIELD for correlating results with input rows. Deserialization errors are returned as DataflowError values. The final result is produced by flushing the deserializer's internal buffer.

Usage

Use RedisLookup when performing lookup joins in Arroyo SQL queries where the lookup table is backed by Redis string values keyed by a text field.

Code Reference

Source Location

Repository: ArroyoSystems_Arroyo
File: crates/arroyo-connectors/src/redis/lookup.rs

Signature

pub struct RedisLookup {
    pub(crate) deserializer: ArrowDeserializer,
    pub(crate) client: RedisClient,
    pub(crate) connection: Option<GeneralConnection>,
    pub(crate) metadata_fields: Vec<MetadataField>,
}

#[async_trait]
impl LookupConnector for RedisLookup {
    fn name(&self) -> String; // "RedisLookup"
    async fn lookup(&mut self, keys: &[ArrayRef])
        -> Option<Result<RecordBatch, DataflowError>>;
}

Import

use arroyo_connectors::redis::lookup::RedisLookup;

I/O Contract

Inputs

Name	Type	Required	Description
keys	&[ArrayRef]	Yes	Exactly one Utf8 array of Redis keys to look up
client	RedisClient	Yes	Standard or Clustered Redis client
metadata_fields	Vec<MetadataField>	No	Metadata fields to include (currently supports "key")

Outputs

Name	Type	Description
result	Option<Result<RecordBatch, DataflowError>>	Deserialized record batch from Redis MGET results, or None if no data

Usage Examples

-- Redis lookup table for SQL JOIN
CREATE TABLE redis_lookup (
    key TEXT PRIMARY KEY,
    value TEXT
) WITH (
    connector = 'redis',
    address = 'redis://localhost:6379',
    type = 'lookup',
    format = 'json'
);

SELECT s.*, r.value
FROM stream_table s
JOIN redis_lookup r ON s.key = r.key;

Related Pages

Environment:ArroyoSystems_Arroyo_Rust_Runtime

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment