Implementation:Bentoml BentoML Arrow Conversion
| Knowledge Sources | |
|---|---|
| Domains | Serialization, Data Conversion |
| Last Updated | 2026-02-13 15:00 GMT |
Overview
Provides bidirectional conversion between Pydantic BaseModel instances and Apache Arrow tables for efficient columnar data serialization.
Description
The arrow module bridges the gap between Pydantic data models and the Apache Arrow columnar format. It converts Pydantic JSON schemas into Arrow schemas by mapping JSON schema field types (integer, number, string, boolean, array, object, tensor, datetime, binary) to their corresponding Arrow data types. The module handles nested structures including lists, structs, maps, nullable fields via anyOf, and tensor types. It also provides functions for serializing a BaseModel instance into an Arrow IPC stream and deserializing an Arrow IPC stream back into a BaseModel instance.
Usage
Use this module when you need to convert BentoML IODescriptor Pydantic models to or from Arrow tables, such as during inter-process communication, batch inference with Spark, or any scenario that benefits from Arrow's efficient columnar storage.
Code Reference
Source Location
- Repository: Bentoml_BentoML
- File: src/_bentoml_impl/arrow.py
- Lines: 1-104
Signature
def model_to_arrow_schema(model: type[BaseModel]) -> pa.Schema: ...
def serialize_to_arrow(model: BaseModel, out_stream: t.BinaryIO) -> None: ...
def deserialize_from_arrow(model: type[T], in_stream: t.BinaryIO) -> T: ...
def is_nullable(field: SchemaDict) -> bool: ...
Import
from _bentoml_impl.arrow import model_to_arrow_schema
from _bentoml_impl.arrow import serialize_to_arrow
from _bentoml_impl.arrow import deserialize_from_arrow
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| model | type[BaseModel] | Yes | A Pydantic model class whose JSON schema is converted to an Arrow schema |
| out_stream | t.BinaryIO | Yes | Binary output stream for Arrow IPC serialization (used by serialize_to_arrow) |
| in_stream | t.BinaryIO | Yes | Binary input stream containing Arrow IPC data (used by deserialize_from_arrow) |
Outputs
| Name | Type | Description |
|---|---|---|
| pa.Schema | pyarrow.Schema | Arrow schema derived from Pydantic model fields (from model_to_arrow_schema) |
| T | BaseModel subclass | Deserialized Pydantic model instance (from deserialize_from_arrow) |
| None | None | serialize_to_arrow writes directly to the output stream |
Usage Examples
import io
from pydantic import BaseModel
from _bentoml_impl.arrow import model_to_arrow_schema, serialize_to_arrow, deserialize_from_arrow
class MyModel(BaseModel):
name: str
score: float
active: bool
# Convert Pydantic model to Arrow schema
schema = model_to_arrow_schema(MyModel)
# schema contains: name: utf8, score: float64, active: bool
# Serialize a model instance to Arrow IPC stream
instance = MyModel(name="test", score=0.95, active=True)
buffer = io.BytesIO()
serialize_to_arrow(instance, buffer)
# Deserialize back from Arrow IPC stream
buffer.seek(0)
restored = deserialize_from_arrow(MyModel, buffer)