Environment:MaterializeInc Materialize Python CI Builder Runtime

Knowledge Sources	MaterializeInc/materialize
Domains	Infrastructure, Build_System
Last Updated	2026-02-08 21:00 GMT

Overview

Python runtime environment within the ci-builder Docker container, providing the `materialize` Python package and all its dependencies for build orchestration, test execution, and release tooling.

Description

This environment defines the Python runtime used by the Materialize build system. All Python tooling runs inside a ci-builder Docker image (with `stable`, `nightly`, and `min` flavors) invoked via `bin/ci-builder run <flavor>`. The Python environment is activated using `bin/pyactivate`, which sets up the virtual environment with all packages from `misc/python/requirements.txt`. The runtime supports both x86_64 and aarch64 architectures and provides Rust cross-compilation toolchains.

Usage

Use this environment for any Python-based build, test, or release operation. It is the foundational prerequisite for all `materialize.*` Python module execution including mzcompose, mzbuild, mkpipeline, and release tooling.

System Requirements

Category	Requirement	Notes
OS	Linux (x86_64 or aarch64)	Inside ci-builder Docker image
Python	Python 3.12+	Provided by ci-builder
Rust	Stable and Nightly toolchains	For compilation; nightly required for sanitizer builds
Hardware	x86_64 targets x86-64-v3 CPU; aarch64 targets neoverse-n1	Specific CPU microarchitectures for optimization

Dependencies

System Packages

`pip` = 25.3
`setuptools` = 80.10.2
Rust stable toolchain (with `x86_64-unknown-linux-gnu` and `aarch64-unknown-linux-gnu` targets)
Rust nightly toolchain (for sanitizer and coverage builds)
`lld` (LLVM linker for Rust builds)
`sccache` (shared compilation cache)
Cross-compilation sysroots at `/opt/x-tools/`

Python Packages (Core)

`psycopg` >= 3.0 (PostgreSQL async driver)
`psycopg2` (PostgreSQL driver, legacy)
`requests` (HTTP client)
`pyyaml` (YAML parser)
`sqlparse` (SQL parser)
`numpy` (numerical computing, used in benchmarks)
`scipy` (scientific computing, used in benchmark termination)
`colored` (terminal color output)

Credentials

`CI_SANITIZER`: Controls which sanitizer to use (`none`, `address`, `hwaddress`, `cfi`, `thread`, `leak`, `undefined`)

Quick Install

# Run inside ci-builder (recommended):
bin/ci-builder run stable bin/pyactivate -m <module>

# Or install Python deps locally:
pip install -r misc/python/requirements-core.txt
pip install -r misc/python/requirements.txt

Code Evidence

CPU target configuration from `xcompile.py:57-68`:

def target_cpu(arch: Arch) -> str:
    """
    Return the CPU micro architecture, assuming a Linux target.
    Sync: This target-cpu should be kept in sync with ci-builder and .cargo/config.
    """
    if arch == Arch.X86_64:
        return "x86-64-v3"
    elif arch == Arch.AARCH64:
        return "neoverse-n1"

Rust flags for cross-compilation from `xcompile.py:116-123`:

rustflags += [
    "-Clink-arg=-Wl,--compress-debug-sections=zlib",
    "-Clink-arg=-Wl,-O3",
    "-Csymbol-mangling-version=v0",
    f"-Ctarget-cpu={_target_cpu}",
    f"-Ctarget-feature={_target_features}",
    "--cfg=tokio_unstable",
]

Coverage flags from `rustc_flags.py:22-25`:

coverage = [
    "-Cinstrument-coverage",
    "-Cllvm-args=-runtime-counter-relocation",
]

Common Errors

Error Message	Cause	Solution
`RustICE` exception	Internal Compiler Error in rustc	Automatic retry configured; see `run_and_detect_rust_ice()` in mzbuild.py
`unknown host architecture`	Running on unsupported CPU arch	Only x86_64 and aarch64 are supported
Missing cross-compilation sysroot	`/opt/x-tools/` not available	Run inside ci-builder container

Compatibility Notes

macOS cross-compilation: Uses Homebrew packages (`lld`, `materializeinc/crosstools`) instead of ci-builder. Bootstrap is version-tracked via a `BOOTSTRAP_VERSION` file.
CPU features: x86_64 enables `+aes`, `+pclmulqdq`; aarch64 enables `+aes`, `+sha2`.
tokio_unstable: The `--cfg=tokio_unstable` flag is always passed to rustc for Tokio nightly features.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment