Implementation:Lance format Lance PackedEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

PackedStructFixedWidthMiniBlockEncoder is a physical encoding that packs multiple fixed-width struct fields into a single row-major buffer for efficient storage and access.

Description

Packed encoding takes struct data where all children are fixed-width and interleaves them in row-major order. For a struct with fields A (4 bytes), B (8 bytes), C (2 bytes), each row is stored as 14 contiguous bytes: [A0][B0][C0][A1][B1][C1]...

The encoding pipeline:

struct_data_block_to_fixed_width_data_block transforms a StructDataBlock into a single FixedWidthDataBlock by interleaving fields row by row.
The interleaved block is compressed using a ValueEncoder (flat mini-block encoding).
The encoding description records per-field bit widths for proper reconstruction.

Decompression reverses this: the PackedStructFixedWidthMiniBlockDecompressor decodes the flat block, then splits each row back into individual field data blocks using a prefix-sum index. The module also provides full-zip variants (PackedStructEncoder, PackedStructDecompressor) for the per-value encoding path.

Usage

Packed encoding is selected for struct columns where all children are fixed-width types and the PACKED_STRUCT_META_KEY metadata is set. It is used for blob descriptors (position/size structs) and other fixed-width struct columns where row-major access is beneficial.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/physical/packed.rs, Lines: 1-1119

Signature

#[derive(Debug, Default)]
pub struct PackedStructFixedWidthMiniBlockEncoder {}

impl MiniBlockCompressor for PackedStructFixedWidthMiniBlockEncoder {
    fn compress(&self, data: DataBlock) -> Result<(MiniBlockCompressed, CompressiveEncoding)>;
}

#[derive(Debug)]
pub struct PackedStructFixedWidthMiniBlockDecompressor {
    bits_per_values: Vec<u64>,
    array_encoding: Box<dyn MiniBlockDecompressor>,
}

impl PackedStructFixedWidthMiniBlockDecompressor {
    pub fn new(description: &PackedStruct) -> Self;
}

Import use lance_encoding::encodings::physical::packed::{PackedStructFixedWidthMiniBlockEncoder, PackedStructFixedWidthMiniBlockDecompressor};

I/O Contract

Direction	Type	Description
Input	`DataBlock::Struct`	Struct data block with fixed-width children
Output	`MiniBlockCompressed`	Row-major interleaved data in mini-block chunks
Output	`CompressiveEncoding`	PackedStruct encoding with per-field bit widths
Output (decompress)	`DataBlock::Struct`	Reconstructed struct data block with individual children

Usage Examples

use lance_encoding::encodings::physical::packed::PackedStructFixedWidthMiniBlockEncoder;
use lance_encoding::encodings::logical::primitive::miniblock::MiniBlockCompressor;

let encoder = PackedStructFixedWidthMiniBlockEncoder::default();
let (compressed, encoding) = encoder.compress(struct_data_block)?;

Related Pages

Lance_format_Lance_StructEncoding - Logical struct encoding that may use packed physical encoding
Lance_format_Lance_ValueEncoding - Used internally for the interleaved buffer
Lance_format_Lance_BlobEncoding - Uses packed encoding for blob descriptors

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment