Implementation:Lance format Lance LegacyPrimitiveEncoding

Knowledge Sources	Lance
Domains	Encoding, Legacy_Format
Last Updated	2026-02-08 19:33 GMT

Overview

The legacy primitive encoding handles fixed-width and simple data types by mapping each field to a single column in the Lance v2.0 format.

Description

⚠️ DEPRECATED: This is legacy code from the Lance v1/v2.0 format, retained only for backward compatibility. See Lance_format_Lance_Warning_Deprecated_Legacy_Encodings.

This module implements primitive field encoding and decoding for the legacy (v2.0) Lance file format. PrimitiveFieldScheduler maps to exactly one column and expects the top-level encoding of each page to be "basic" (an optional validity buffer plus a fixed-width values buffer). It supports booleans, integers, floats, dates, timestamps, decimals, intervals, durations, fixed-size binaries, fixed-size lists of primitives, and string/binary types. The scheduler handles page navigation by maintaining a list of PrimitivePage entries and skipping empty pages (which could exist from buggy older writers). PrimitiveFieldEncoder accumulates arrays using an AccumulationQueue and delegates per-page encoding to an ArrayEncodingStrategy. Optional data validation is supported for verifying decoded arrays.

Usage

Use this encoding for all fixed-width and primitive-like fields. The CoreFieldEncodingStrategy selects PrimitiveFieldEncoder for booleans, numerics, timestamps, fixed-size binary, fixed-size lists of primitives, and string/binary types. During reading, PrimitiveFieldScheduler is constructed from column page metadata and buffer information.

Code Reference

Source Location

rust/lance-encoding/src/previous/encodings/logical/primitive.rs

Signature

pub struct PrimitiveFieldScheduler {
    data_type: DataType,
    page_schedulers: Vec<PrimitivePage>,
    num_rows: u64,
    should_validate: bool,
    column_index: u32,
}

impl PrimitiveFieldScheduler {
    pub fn new(
        column_index: u32,
        data_type: DataType,
        pages: Arc<[PageInfo]>,
        buffers: ColumnBuffers,
        should_validate: bool,
    ) -> Self;
}

pub struct PrimitiveFieldEncoder { /* fields omitted */ }

impl PrimitiveFieldEncoder {
    pub fn try_new(
        options: &EncodingOptions,
        array_encoding_strategy: Arc<dyn ArrayEncodingStrategy>,
        column_index: u32,
        field: Field,
    ) -> Result<Self>;
}

Import

use lance_encoding::previous::encodings::logical::primitive::{
    PrimitiveFieldScheduler, PrimitiveFieldEncoder, PrimitiveFieldDecoder,
};

I/O Contract

Input	Type	Description
column_index	u32	Column index in the file
data_type	DataType	Arrow data type for the field
pages	Arc<[PageInfo]>	Page metadata including encoding and buffer offsets
buffers	ColumnBuffers	Buffer positions for the column
arrays	ArrayRef	Arrow arrays to encode

Output	Type	Description
decoded	ArrayRef	Decoded Arrow array matching the field data type
encoded_pages	Vec<EncodedPage>	Encoded pages with data buffers and encoding descriptors
encoded_column	EncodedColumn	Complete encoded column with all pages

Usage Examples

use lance_encoding::previous::encodings::logical::primitive::PrimitiveFieldScheduler;
use lance_encoding::decoder::{ColumnBuffers, PageInfo};
use arrow_schema::DataType;
use std::sync::Arc;

// Create a primitive field scheduler for a Float64 column
let pages: Arc<[PageInfo]> = /* from file metadata */;
let buffers: ColumnBuffers = /* from file metadata */;
let scheduler = PrimitiveFieldScheduler::new(
    0,                    // column index
    DataType::Float64,    // data type
    pages,
    buffers,
    false,                // should_validate
);

// Schedule ranges
let ranges = vec![0..500];
let filter = FilterExpression::no_filter();
let mut job = scheduler.schedule_ranges(&ranges, &filter)?;

Related Pages

• Lance_format_Lance_LegacyEncoder - Encoding strategy that creates primitive encoders
• Lance_format_Lance_LegacyDecoder - Base decoder traits
• Lance_format_Lance_LegacyPhysicalDispatch - Dispatches to physical page schedulers
• Lance_format_Lance_LegacyBasicEncoding - Physical encoding used by primitives
• Lance_format_Lance_LegacyBitpackEncoding - Optional bitpacking for integer primitives
• Heuristic:Lance_format_Lance_Warning_Deprecated_Legacy_Encodings