Principle:Eventual Inc Daft Rust Python Bridge
| Knowledge Sources | |
|---|---|
| Domains | FFI, Rust_Python_Bridge |
| Last Updated | 2026-02-08 14:00 GMT |
Overview
Architectural pattern that bridges a Rust computation engine to a Python API layer via PyO3 module registration, producing a native extension that Python loads transparently.
Description
The Rust-Python Bridge uses PyO3's `#[pymodule]` and `#[pyfunction]` macros to expose Rust types, functions, and sub-crate modules as a Python native extension (`.so` or `.pyd`). The root module entry point initializes global state (allocator, logging, telemetry, function registry) and registers all sub-crate modules in a single pass. This produces a `daft.daft` extension module that the Python `daft` package imports to access the Rust engine.
Usage
Apply this principle when building a Python library backed by a Rust computation engine. The pattern is standard for data processing libraries (polars, datafusion-python, daft) that need Python ergonomics with Rust performance.
Theoretical Basis
The bridge architecture has three layers:
- Rust engine: Pure Rust crates implementing core logic (query planning, execution, I/O).
- PyO3 binding: A `cdylib` crate that wraps Rust types with `#[pyclass]` and functions with `#[pyfunction]`, registers sub-modules, and initializes global state.
- Python API: Pure Python modules that import from the native extension and provide a Pythonic interface (DataFrame, Expression, etc.).
The `maturin` build tool automates the compilation and packaging of this three-layer system into a standard Python wheel.