Implementation:Lance format Lance PrimitiveEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

PrimitiveStructuralEncoder is the core logical encoding that handles leaf-level primitive, fixed-width, and variable-width data through either miniblock or full-zip structural encoding strategies.

Description

This is the largest and most central encoding module in Lance. It handles the final stage of logical encoding where data blocks are compressed and organized into pages. The encoder supports two structural encoding modes:

Miniblock: Data is compressed into small chunks (up to ~8KiB each, up to 4096 values). Chunks are independently decodable, enabling efficient random access. This is the default for most data types.
Full-zip: Compressed buffers are zipped together so all parts of a value are stored contiguously. Requires transparent compression and is best for large data types.

The module includes the StructuralPrimitiveFieldScheduler for scheduling page reads, which manages page metadata caching, range scheduling, and shard-based I/O. It also contains StructuralPrimitiveFieldDecoder for decoding pages back into Arrow arrays using rep/def unraveling. The encoder uses an AccumulationQueue to batch data before compression, and supports dictionary encoding selection based on cardinality ratios.

Usage

This encoding is used for all leaf-level columns: integers, floats, strings, binary data, fixed-size binary, booleans, and dictionary-encoded types. It is the terminal encoder in the encoding chain -- list, struct, map, and FSL encoders all eventually delegate to a primitive encoder for their leaf data.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/logical/primitive.rs, Lines: 1-6514

Signature

pub struct PrimitiveStructuralEncoder {
    // Internal fields: compression_strategy, column_index, field,
    // accumulation_queue, rows_in_flight, encoding_mode, etc.
}

impl PrimitiveStructuralEncoder {
    pub fn try_new(
        options: &EncodingOptions,
        compression_strategy: Arc<dyn CompressionStrategy>,
        column_index: u32,
        field: Field,
        field_metadata: Arc<HashMap<String, String>>,
    ) -> Result<Self>;
}

pub struct StructuralPrimitiveFieldScheduler {
    // Manages page schedulers, metadata caching, column info
}

pub struct StructuralPrimitiveFieldDecoder {
    // Decodes pages into Arrow arrays using rep/def
}

Import use lance_encoding::encodings::logical::primitive::PrimitiveStructuralEncoder;

I/O Contract

Direction	Type	Description
Input	`ArrayRef`	Any leaf-level Arrow array (primitive, string, binary, etc.)
Input	`RepDefBuilder`	Serialized repetition/definition levels
Input	`CompressionStrategy`	Strategy that selects physical compressors
Output	`Vec<EncodeTask>`	Async tasks producing encoded pages
Output	`EncodedColumn`	Column metadata, page descriptions, and compressed data
Output (decode)	`DecodedArray`	Reconstructed Arrow array with nullability

Usage Examples

use lance_encoding::encodings::logical::primitive::PrimitiveStructuralEncoder;
use lance_core::datatypes::Field;
use std::collections::HashMap;
use std::sync::Arc;

let encoder = PrimitiveStructuralEncoder::try_new(
    &encoding_options,
    compression_strategy,
    0, // column_index
    field,
    Arc::new(HashMap::new()),
)?;

Related Pages

Lance_format_Lance_MiniBlockCompressor - Miniblock compression trait
Lance_format_Lance_FullZipCompressor - Full-zip compression trait
Lance_format_Lance_ValueEncoding - Default value encoder for fixed-width data
Lance_format_Lance_BinaryEncoding - Binary encoder for variable-width data
Lance_format_Lance_ConstantEncoding - Constant page encoding for uniform values

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment