Environment:Lance format Lance Python Environment
| Knowledge Sources | |
|---|---|
| Domains | Infrastructure, Python_Bindings |
| Last Updated | 2026-02-08 19:00 GMT |
Overview
Python 3.9+ environment with PyArrow >= 14, NumPy >= 1.22, and maturin >= 1.4 build system for the Lance Python bindings (pylance).
Description
The Lance Python package (`pylance`) provides Python bindings to the Rust Lance library via PyO3. Building from source requires maturin >= 1.4 and Rust 1.90.0. The binary wheels support Python 3.9 through 3.13 using the stable ABI (`abi3-py39`). Runtime requires only PyArrow >= 14 and NumPy >= 1.22 as core dependencies.
Usage
This environment is required for any Python-based interaction with Lance datasets: reading, writing, scanning, vector search, full-text search, and all Python API operations. It is also needed for running the Python test suite and benchmarks.
System Requirements
| Category | Requirement | Notes |
|---|---|---|
| OS | Linux, macOS, or Windows | TensorFlow extras Linux-only |
| Hardware | x86_64 or aarch64 CPU | Wheels built for both architectures |
| Disk | 5GB+ SSD | For virtual environment and build artifacts |
Dependencies
Core Runtime
- `pyarrow` >= 14 — Apache Arrow Python bindings
- `numpy` >= 1.22 — Numerical computing
- `lance-namespace` >= 0.4.5 — Lance namespace package
Build System
- `maturin` >= 1.4 — Rust-Python build tool
- Rust 1.90.0 toolchain (see Environment:Lance_format_Lance_Rust_Toolchain)
Optional: Tests
- `boto3` — AWS SDK for integration tests
- `datasets` — Hugging Face datasets
- `duckdb` — DuckDB integration
- `ml_dtypes` — ML data types
- `pillow` — Image processing
- `pandas` — Data manipulation
- `polars[pyarrow,pandas]` — Polars dataframe
- `pytest` — Test framework
- `datafusion` >= 50.1 — DataFusion SQL engine
- `tensorflow` — Linux only, will be deprecated
Optional: PyTorch
- `torch` >= 2.0 — PyTorch deep learning framework
Optional: Geospatial
- `geoarrow-rust-core` — GeoArrow Rust bindings
- `geoarrow-rust-io` — GeoArrow I/O utilities
Credentials
The following environment variables configure Python-side behaviour:
- `LANCE_LOG` — Controls logging level (trace, debug, info, warn, error)
- `LANCE_LOG_FILE` — Path to log file output
- `LANCE_LOG_TS_PRECISION` — Timestamp precision (ns, us, ms, s)
- `LANCE_MEM_POOL_SIZE` — Memory pool size in bytes
- `LANCE_BYPASS_SPILLING` — Flag to bypass memory spilling (0 or 1)
- `LANCE_DEFAULT_BATCH_SIZE` — Default batch size for operations
- `LANCE_LANGUAGE_MODEL_HOME` — Directory for tokenizer language models
Quick Install
# Install from PyPI (pre-built wheel)
pip install pylance
# Or build from source
pip install maturin>=1.4
maturin develop
# Install with test dependencies
pip install -e '.[tests]'
# Install with PyTorch support
pip install -e '.[torch]'
Code Evidence
Python version requirement from `python/pyproject.toml:10`:
requires-python = ">=3.9"
Core dependencies from `python/pyproject.toml:4`:
dependencies = ["pyarrow>=14", "numpy>=1.22", "lance-namespace>=0.4.5"]
Build system from `python/pyproject.toml:45-47`:
[build-system]
requires = ["maturin>=1.4"]
build-backend = "maturin"
PyO3 ABI3 binding from `python/Cargo.toml` (pyo3 features):
pyo3 = { version = "0.26", features = ["extension-module", "abi3-py39", "py-clone", "chrono"] }
Common Errors
| Error Message | Cause | Solution |
|---|---|---|
| `ImportError: PyArrow >= 14 is required` | PyArrow version too old | `pip install 'pyarrow>=14'` |
| `maturin: command not found` | maturin not installed for source build | `pip install maturin>=1.4` |
| `error: can't find Rust compiler` | Rust not installed for source build | Install Rust via rustup (see Rust_Toolchain environment) |
| `ModuleNotFoundError: No module named 'lance'` | Extension not built after Rust changes | Run `maturin develop` to rebuild |
Compatibility Notes
- Windows: Subprocess-based tests and tracing tests are skipped. `os.fork()` not available.
- Linux only: TensorFlow integration tests require Linux. Will be deprecated.
- Python 3.9: Minimum supported via ABI3 stable interface. Tested in CI.
- Python 3.13: Maximum tested version. Used for pyright type checking.