Implementation:Lance format Lance EncodingFormat

Knowledge Sources	Lance
Domains	Encoding, Columnar_Data
Last Updated	2026-02-08 19:33 GMT

Overview

The EncodingFormat module provides protobuf message definitions and utility builders (ProtobufUtils, ProtobufUtils21) for describing Lance page and column encodings in file metadata.

Description

This module is the bridge between Lance's in-memory encoding decisions and the persistent metadata stored in Lance files. It contains two sets of protobuf definitions and corresponding utility structs:

1. Legacy Format (pb module, ProtobufUtils):

Used by Lance 2.0 files. Key encoding messages include:

• ArrayEncoding -- Top-level encoding descriptor with variants for Flat, Nullable, FixedSizeList, Bitpacked, BitpackedForNonNeg, InlineBitpacking, OutOfLineBitpacking, Variable, Fsst, Rle, ByteStreamSplit, Binary, Dictionary, PackedStruct, Block, Constant, and more.
• ColumnEncoding -- Column-level metadata (e.g., Values or Blob).
• Buffer -- Refers to page or column buffers by index.

2. Structural Format (pb21 module, ProtobufUtils21):

Used by Lance 2.1+ files. Key encoding messages include:

• CompressiveEncoding -- A unified compression descriptor with variants: Flat, Constant, FixedSizeList, Variable, InlineBitpacking, OutOfLineBitpacking, General (LZ4/Zstd wrapper), Rle, ByteStreamSplit, Fsst.
• PageLayout -- Structural page layout (miniblock, fullzip, or blob).
• RepDefLayer -- Describes each layer of rep/def interpretation.

Utility Builders:

Both ProtobufUtils and ProtobufUtils21 provide static factory methods to construct complex protobuf messages. This avoids verbose nested constructor calls throughout the codebase. The impl_common_protobuf_utils! macro generates shared methods across both format versions for flat, constant, fsl, variable, inline_bitpacking, out_of_line_bitpacking, buffer_compression, wrapped, rle, byte_stream_split, and fsst.

Usage

Use this module when:

• Writing encoding metadata to a Lance file
• Reading encoding metadata to reconstruct decoders
• Implementing a new encoding that needs a protobuf descriptor

Code Reference

Source Location	rust/lance-encoding/src/format.rs
Protobuf Modules	pb (2.0 format), pb21 (2.1+ format)
Utility Structs	ProtobufUtils (2.0), ProtobufUtils21 (2.1+)
Import	use lance_encoding::format::{pb, pb21, ProtobufUtils, ProtobufUtils21};

I/O Contract

ProtobufUtils (2.0) Factory Methods:

Method	Output	Description
flat_encoding(bits, buf_idx, compression)	ArrayEncoding	Fixed-width flat encoding
basic_no_null_encoding(values)	ArrayEncoding	Non-nullable wrapper
basic_some_null_encoding(validity, values)	ArrayEncoding	Nullable wrapper with validity
bitpacked_encoding(compressed, uncompressed, buf, signed)	ArrayEncoding	Bitpacked encoding
dict_encoding(indices, items, num_items)	ArrayEncoding	Dictionary encoding
binary(indices, bytes, null_adj)	ArrayEncoding	Binary/string encoding
rle(bits_per_value)	ArrayEncoding	Run-length encoding

ProtobufUtils21 (2.1+) Factory Methods:

Method	Output	Description
flat(bits, compression)	CompressiveEncoding	Fixed-width flat
constant(value)	CompressiveEncoding	Constant value
variable(offsets, values_compression)	CompressiveEncoding	Variable-width
rle(values, run_lengths)	CompressiveEncoding	Run-length encoding
fsst(data, symbol_table)	CompressiveEncoding	FSST string compression
wrapped(compression, values)	Result<CompressiveEncoding>	General compression wrapper
miniblock_layout(rep_def, value_desc, chunks)	PageLayout	Miniblock page layout
fullzip_layout(rep_def, value_desc, rows_per_chunk)	PageLayout	Fullzip page layout

Usage Examples

use lance_encoding::format::{ProtobufUtils, ProtobufUtils21};

// Create a 2.0 format flat encoding descriptor
let encoding = ProtobufUtils::flat_encoding(
    32,    // bits per value
    0,     // buffer index
    None,  // no compression
);

// Create a 2.1 format flat encoding descriptor
let encoding_21 = ProtobufUtils21::flat(32, None);

// Create a 2.1 RLE descriptor wrapping a flat value encoding
let rle_encoding = ProtobufUtils21::rle(
    ProtobufUtils21::flat(32, None),         // values
    ProtobufUtils21::flat(8, None),          // run lengths
);

Related Pages

• Lance_format_Lance_CoreEncoder - Encoders use ProtobufUtils/ProtobufUtils21 to describe their output
• Lance_format_Lance_BatchDecodeStream - Decoders read protobuf descriptors to reconstruct data
• Lance_format_Lance_Compression_Traits - Compression strategies produce CompressiveEncoding descriptors
• Lance_format_Lance_RepDef - DefinitionInterpretation maps to RepDefLayer protobuf values