Implementation:Lance format Lance PrimitiveEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

PrimitiveStructuralEncoder is the core logical encoding that handles leaf-level primitive, fixed-width, and variable-width data through either miniblock or full-zip structural encoding strategies.

Description

This is the largest and most central encoding module in Lance. It handles the final stage of logical encoding where data blocks are compressed and organized into pages. The encoder supports two structural encoding modes:

• Miniblock: Data is compressed into small chunks (up to ~8KiB each, up to 4096 values). Chunks are independently decodable, enabling efficient random access. This is the default for most data types.
• Full-zip: Compressed buffers are zipped together so all parts of a value are stored contiguously. Requires transparent compression and is best for large data types.

The module includes the StructuralPrimitiveFieldScheduler for scheduling page reads, which manages page metadata caching, range scheduling, and shard-based I/O. It also contains StructuralPrimitiveFieldDecoder for decoding pages back into Arrow arrays using rep/def unraveling. The encoder uses an AccumulationQueue to batch data before compression, and supports dictionary encoding selection based on cardinality ratios.

Usage

This encoding is used for all leaf-level columns: integers, floats, strings, binary data, fixed-size binary, booleans, and dictionary-encoded types. It is the terminal encoder in the encoding chain -- list, struct, map, and FSL encoders all eventually delegate to a primitive encoder for their leaf data.

Code Reference

Source Location	Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/logical/primitive.rs, Lines: 1-6514
Signature	pub struct PrimitiveStructuralEncoder { // Internal fields: compression_strategy, column_index, field, // accumulation_queue, rows_in_flight, encoding_mode, etc. } impl PrimitiveStructuralEncoder { pub fn try_new( options: &EncodingOptions, compression_strategy: Arc<dyn CompressionStrategy>, column_index: u32, field: Field, field_metadata: Arc<HashMap<String, String>>, ) -> Result<Self>; } pub struct StructuralPrimitiveFieldScheduler { // Manages page schedulers, metadata caching, column info } pub struct StructuralPrimitiveFieldDecoder { // Decodes pages into Arrow arrays using rep/def }
Import	use lance_encoding::encodings::logical::primitive::PrimitiveStructuralEncoder;

I/O Contract

Direction	Type	Description
Input	ArrayRef	Any leaf-level Arrow array (primitive, string, binary, etc.)
Input	RepDefBuilder	Serialized repetition/definition levels
Input	CompressionStrategy	Strategy that selects physical compressors
Output	Vec<EncodeTask>	Async tasks producing encoded pages
Output	EncodedColumn	Column metadata, page descriptions, and compressed data
Output (decode)	DecodedArray	Reconstructed Arrow array with nullability

Usage Examples

use lance_encoding::encodings::logical::primitive::PrimitiveStructuralEncoder;
use lance_core::datatypes::Field;
use std::collections::HashMap;
use std::sync::Arc;

let encoder = PrimitiveStructuralEncoder::try_new(
    &encoding_options,
    compression_strategy,
    0, // column_index
    field,
    Arc::new(HashMap::new()),
)?;

Related Pages

• Implementation:Lance_format_Lance_MiniBlockCompressor - Miniblock compression trait
• Implementation:Lance_format_Lance_FullZipCompressor - Full-zip compression trait
• Implementation:Lance_format_Lance_ValueEncoding - Default value encoder for fixed-width data
• Implementation:Lance_format_Lance_BinaryEncoding - Binary encoder for variable-width data
• Implementation:Lance_format_Lance_ConstantEncoding - Constant page encoding for uniform values