Implementation:ArroyoSystems Arroyo Json Schema Module

Overview

JSON Schema Module provides functions for converting between Arrow field types and JSON Schema / Kafka Connect schema formats. It serves as the bridge between Arroyo's internal Arrow-based type system and external JSON-based schema specifications.

Description

The module exports four primary functions:

• field_to_json_schema: Maps a single Arrow Field to a JSON Schema value object. Handles nulls, booleans, integers, floats, decimals, timestamps (as date-time strings), binary data (as integer arrays), strings, lists, and nested structs.

• arrow_to_json_schema: Converts Arrow Fields into a complete JSON Schema object with type: "object", properties, and required arrays. Non-nullable fields are listed in the required array.

• field_to_kafka_json: Maps a single Arrow Field to a Kafka Connect JSON schema format, which uses a bespoke schema representation with type, field, optional, and optional name fields. Timestamps get the org.apache.kafka.connect.data.Timestamp name annotation, dates get org.apache.kafka.connect.data.Date, and decimals get org.apache.kafka.connect.data.Decimal with a scale parameter.

• arrow_to_kafka_json: Converts Arrow Fields into a complete Kafka Connect JSON schema with type: "struct".

The module also re-exports the encoders and schema submodules.

Usage

These functions are called by ArrowSerializer to generate schema metadata for JSON and Kafka Connect sinks.

Code Reference

Source Location

crates/arroyo-formats/src/json/mod.rs

Signature

pub fn field_to_json_schema(field: &Field) -> Value

pub fn arrow_to_json_schema(fields: &Fields) -> Value

pub fn field_to_kafka_json(field: &Field) -> Value

pub fn arrow_to_kafka_json(name: &str, fields: &Fields) -> Value

Import

use arroyo_formats::json::{arrow_to_json_schema, arrow_to_kafka_json};

I/O Contract

Inputs

Name	Type	Description
field	&Field	Arrow field definition to convert
fields	&Fields	Arrow field collection to convert
name	&str	Schema name for Kafka Connect format

Outputs

Name	Type	Description
json_schema	Value	JSON Schema object or Kafka Connect schema object

Usage Examples

use arrow_schema::{Field, DataType, Fields};

let fields = Fields::from(vec![
    Field::new("name", DataType::Utf8, false),
    Field::new("age", DataType::Int32, true),
]);

// Generate JSON Schema
let json_schema = arrow_to_json_schema(&fields);
// {"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}, "required": ["name"]}

// Generate Kafka Connect schema
let kafka_schema = arrow_to_kafka_json("MyRecord", &fields);
// {"type": "struct", "name": "MyRecord", "fields": [...], "optional": false}

Related Pages

• ArroyoSystems_Arroyo_Json_Schema_Converter - JSON Schema to Arrow conversion
• ArroyoSystems_Arroyo_Json_Encoders - Custom JSON encoding
• ArroyoSystems_Arroyo_Format_Serializer - Uses these schemas for output formatting