Implementation:Lance format Lance LegacyEncoder

Knowledge Sources	Lance
Domains	Encoding, Legacy_Format
Last Updated	2026-02-08 19:33 GMT

Overview

The legacy encoder module provides the core traits and strategy implementations for encoding Arrow data into the Lance v2.0 file format.

Description

⚠️ DEPRECATED: This is legacy code from the Lance v1/v2.0 format, retained only for backward compatibility. See Lance_format_Lance_Warning_Deprecated_Legacy_Encodings.

This module defines the foundational encoding infrastructure for the legacy (v2.0) Lance file format. It contains the ArrayEncoder trait for physical array encoding, the ArrayEncodingStrategy trait for selecting encoders per page, and the CoreFieldEncodingStrategy / CoreArrayEncodingStrategy implementations that orchestrate encoding decisions across all supported data types. The strategy selects among encoders including bitpacking, FSST, dictionary encoding, binary encoding, and packed struct encoding based on data characteristics such as cardinality, data size, and type. It also defines the EncodedArray struct which pairs encoded data buffers with their protobuf encoding descriptions. These components support writing data in older Lance file format versions.

Usage

Use this module when writing Lance files in the v2.0 format. CoreFieldEncodingStrategy is the main entry point: instantiate it with a target LanceFileVersion and call create_field_encoder to obtain the appropriate FieldEncoder for each field in the schema. CoreArrayEncodingStrategy is used internally to select physical encoders for individual data pages.

Code Reference

Source Location

rust/lance-encoding/src/previous/encoder.rs

Signature

// Core encoding trait
pub trait ArrayEncoder: std::fmt::Debug + Send + Sync {
    fn encode(
        &self,
        data: DataBlock,
        data_type: &DataType,
        buffer_index: &mut u32,
    ) -> Result<EncodedArray>;
}

// Strategy for choosing an encoder per page
pub trait ArrayEncodingStrategy: Send + Sync + std::fmt::Debug {
    fn create_array_encoder(
        &self,
        arrays: &[ArrayRef],
        field: &Field,
    ) -> Result<Box<dyn ArrayEncoder>>;
}

// Main field encoding strategy
pub struct CoreFieldEncodingStrategy {
    pub array_encoding_strategy: Arc<dyn ArrayEncodingStrategy>,
    pub version: LanceFileVersion,
}

impl CoreFieldEncodingStrategy {
    pub fn new(version: LanceFileVersion) -> Self;
}

Import

use lance_encoding::previous::encoder::{
    ArrayEncoder, ArrayEncodingStrategy, CoreFieldEncodingStrategy, EncodedArray,
};

I/O Contract

Input	Type	Description
data	DataBlock	Raw data block to encode
data_type	&DataType	Arrow data type of the input
buffer_index	&mut u32	Mutable index tracking buffer positions
arrays	&[ArrayRef]	Sample arrays for statistics-based encoding decisions
field	&Field	Lance field descriptor with metadata

Output	Type	Description
encoded	EncodedArray	Encoded data buffers paired with protobuf encoding descriptor
field_encoder	Box<dyn FieldEncoder>	Configured field encoder for a given schema field

Usage Examples

use lance_encoding::previous::encoder::CoreFieldEncodingStrategy;
use lance_encoding::encoder::{FieldEncodingStrategy, ColumnIndexSequence, EncodingOptions};
use lance_encoding::version::LanceFileVersion;
use lance_core::datatypes::Field;

// Create a strategy for v2.0 format
let strategy = CoreFieldEncodingStrategy::new(LanceFileVersion::V2_0);

// Use the strategy to create a field encoder
let field: Field = /* obtain from schema */;
let mut col_index = ColumnIndexSequence::default();
let options = EncodingOptions::default();
let encoder = strategy.create_field_encoder(&strategy, &field, &mut col_index, &options)?;

Related Pages

• Lance_format_Lance_LegacyDecoder - Corresponding legacy decoding infrastructure
• Lance_format_Lance_LegacyPrimitiveEncoding - Primitive field encoder used by the strategy
• Lance_format_Lance_LegacyListEncoding - List field encoder used by the strategy
• Lance_format_Lance_LegacyBlobEncoding - Blob field encoder used by the strategy
• Lance_format_Lance_LegacyStructEncoding - Struct field encoder used by the strategy
• Lance_format_Lance_LegacyPhysicalDispatch - Physical encoding dispatch
• Heuristic:Lance_format_Lance_Warning_Deprecated_Legacy_Encodings