Implementation:ArroyoSystems Arroyo Avro Schema Converter

Overview

Avro Schema Converter provides bidirectional conversion between Apache Avro schemas and Arrow schemas. It maps Avro record types to Arrow struct types and vice versa, handling logical types, nullable unions, nested records, arrays, and decimal types.

Description

The module contains two primary conversion directions:

• to_avro: Converts Arrow Fields into an Avro Schema (record type). Each Arrow field is mapped to an Avro field via field_to_avro and arrow_to_avro. Nullable Arrow fields are wrapped in Avro union types ["null", type]. Supports:
  • Primitive types (boolean, int, long, float, double, string, bytes)
  • Timestamps with logical type annotations (timestamp-micros, timestamp-millis, local-timestamp-*)
  • Dates (date logical type on int)
  • Decimals (decimal logical type on bytes with precision/scale)
  • Lists/arrays (recursive)
  • Nested structs/records (recursive)

• to_arrow: Parses an Avro schema string and converts it to an Arrow Schema. Uses to_arrow_datatype for recursive type mapping. Avro unions with exactly one null and one non-null variant are mapped to nullable Arrow fields. Multi-variant unions are mapped to Utf8 with JSON extension metadata. Supports:
  • All primitive Avro types
  • Logical types: TimeMillis, TimeMicros, TimestampMillis, TimestampMicros, LocalTimestamp*
  • Records (mapped to Struct)
  • Enums, UUIDs (mapped to Utf8)
  • Complex unions (mapped to JSON-typed Utf8)

Usage

These functions are used during table creation to translate user-provided Avro schemas into Arrow schemas for internal processing, and during serialization to generate Avro schemas from Arrow schemas.

Code Reference

Source Location

crates/arroyo-formats/src/avro/schema.rs

Signature

pub fn to_avro(name: &str, fields: &Fields) -> Schema

pub fn to_arrow(schema: &str) -> anyhow::Result<arrow_schema::Schema>

Import

use arroyo_formats::avro::schema::{to_avro, to_arrow};

I/O Contract

Inputs

Name	Type	Description
name	&str	Record name for the generated Avro schema
fields	&Fields	Arrow field definitions to convert to Avro
schema	&str	Avro schema JSON string to convert to Arrow

Outputs

Name	Type	Description
avro_schema	Schema	Apache Avro schema (record type)
arrow_schema	arrow_schema::Schema	Arrow schema with mapped data types

Usage Examples

// Convert Arrow fields to Avro schema
let avro_schema = to_avro("MyRecord", &arrow_schema.fields);

// Convert Avro schema string to Arrow schema
let avro_json = r#"{"type": "record", "name": "User", "fields": [
    {"name": "name", "type": "string"},
    {"name": "age", "type": "int"}
]}"#;
let arrow_schema = to_arrow(avro_json)?;

Related Pages

• ArroyoSystems_Arroyo_Avro_Deserializer - Uses Arrow schemas for deserialization
• ArroyoSystems_Arroyo_Avro_Serializer - Uses Avro schemas for serialization
• ArroyoSystems_Arroyo_Json_Schema_Converter - Similar conversion for JSON schemas
• ArroyoSystems_Arroyo_Proto_Schema_Converter - Similar conversion for Protobuf schemas