Implementation:Lance format Lance LegacyPackedStructEncoding

Knowledge Sources	Lance
Domains	Encoding, Legacy_Format
Last Updated	2026-02-08 19:33 GMT

Overview

The legacy packed struct encoding stores struct fields in row-major (interleaved) layout for efficient access of complete rows in the Lance v2.0 format.

Description

⚠️ DEPRECATED: This is legacy code from the Lance v1/v2.0 format, retained only for backward compatibility. See Lance_format_Lance_Warning_Deprecated_Legacy_Encodings.

This module implements the packed struct physical encoding for the legacy (v2.0) Lance file format. PackedStructPageScheduler reads contiguous blocks of row-interleaved bytes from a single buffer, where each row's fields are concatenated together (e.g., for fields x, y, z with 1, 4, 8 bytes respectively, each row occupies 13 consecutive bytes). During decoding, PackedStructPageDecoder de-interleaves the packed bytes by iterating over rows and extracting each field's bytes at the correct stride offset, producing a StructDataBlock with separate FixedWidthDataBlock children. The PackedStructEncoder performs the reverse operation: it interleaves child field data into a single packed buffer and delegates the actual byte encoding to an inner ArrayEncoder. This encoding is selected for struct fields marked with the PACKED_STRUCT_META_KEY metadata.

Usage

Use this encoding for struct fields whose children are all fixed-width primitive types and where row-oriented access patterns are common. The CoreFieldEncodingStrategy selects this encoding when the struct field metadata contains either PACKED_STRUCT_META_KEY or PACKED_STRUCT_LEGACY_META_KEY. During reading, PackedStructPageScheduler is created by the physical dispatch from PackedStruct protobuf encoding.

Code Reference

Source Location

rust/lance-encoding/src/previous/encodings/physical/packed_struct.rs

Signature

pub struct PackedStructPageScheduler {
    _inner_schedulers: Vec<Box<dyn PageScheduler>>,
    fields: Fields,
    buffer_offset: u64,
}

impl PackedStructPageScheduler {
    pub fn new(
        _inner_schedulers: Vec<Box<dyn PageScheduler>>,
        struct_datatype: DataType,
        buffer_offset: u64,
    ) -> Self;
}

pub struct PackedStructEncoder {
    inner_encoder: Box<dyn ArrayEncoder>,
    fields: Fields,
}

impl PackedStructEncoder {
    pub fn new(inner_encoder: Box<dyn ArrayEncoder>, fields: Fields) -> Self;
}

Import

use lance_encoding::previous::encodings::physical::packed_struct::{
    PackedStructPageScheduler, PackedStructEncoder,
};

I/O Contract

Input	Type	Description
struct_datatype	DataType	Must be DataType::Struct with fixed-width children
buffer_offset	u64	Offset of the packed buffer in the file
ranges	&[Range<u64>]	Row ranges to decode
data	DataBlock	Struct data block with fixed-width children to encode

Output	Type	Description
decoded	DataBlock::Struct	Struct data block with de-interleaved child fields
encoded	EncodedArray	Interleaved byte buffer with encoding descriptor

Usage Examples

use lance_encoding::previous::encodings::physical::packed_struct::PackedStructPageScheduler;
use lance_encoding::decoder::PageScheduler;
use arrow_schema::{DataType, Field, Fields};

// Define a struct with fixed-width fields
let fields = Fields::from(vec![
    Field::new("x", DataType::Float32, false),
    Field::new("y", DataType::Float32, false),
    Field::new("z", DataType::Float32, false),
]);

// Create the scheduler
let inner_schedulers: Vec<Box<dyn PageScheduler>> = /* from inner encodings */;
let scheduler = PackedStructPageScheduler::new(
    inner_schedulers,
    DataType::Struct(fields),
    0,    // buffer_offset
);

// Schedule ranges for reading
let ranges = vec![0..100];
let io: Arc<dyn EncodingsIo> = /* from context */;
let decoder_fut = scheduler.schedule_ranges(&ranges, &io, 0);

Related Pages

• Lance_format_Lance_LegacyPhysicalDispatch - Creates PackedStructPageScheduler from protobuf
• Lance_format_Lance_LegacyStructEncoding - Regular (non-packed) struct encoding
• Lance_format_Lance_LegacyEncoder - Strategy that selects packed struct encoding
• Lance_format_Lance_LegacyValueEncoding - Inner encoder for the packed byte buffer
• Heuristic:Lance_format_Lance_Warning_Deprecated_Legacy_Encodings