Implementation:Lance format Lance BinaryEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

BinaryMiniBlockEncoder is a physical encoding for variable-width data (strings, binary) that chunks offset-based data into mini-block format with configurable chunk sizes.

Description

This module provides the leaf-level encoding for variable-width data. The BinaryMiniBlockEncoder takes variable-width data blocks (offsets + data buffers) and organizes them into mini-block chunks. Each chunk contains localized offsets rebased to the chunk start, followed by the corresponding data bytes, padded to alignment boundaries.

Key implementation details:

• Default target chunk size is 4 KiB, configurable via the LANCE_BINARY_MINIBLOCK_CHUNK_SIZE environment variable.
• The chunk_offsets function handles both i32 and i64 offset types and produces aligned chunks.
• Each chunk contains (num_values + 1) offsets followed by the data bytes, padded to the required alignment.
• Non-last chunks contain a power-of-2 number of values.
• The module also provides BinaryBlockDecompressor and VariableEncoder for decompression and full-zip paths.

Usage

Used automatically for encoding Utf8, Binary, LargeUtf8, and LargeBinary columns. It is the default physical encoding for all variable-width leaf data.

Code Reference

Source Location	Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/physical/binary.rs, Lines: 1-973
Signature	#[derive(Debug)] pub struct BinaryMiniBlockEncoder { minichunk_size: i64, } impl Default for BinaryMiniBlockEncoder { fn default() -> Self; } impl MiniBlockCompressor for BinaryMiniBlockEncoder { fn compress(&self, page: DataBlock) -> Result<(MiniBlockCompressed, CompressiveEncoding)>; } pub struct BinaryBlockDecompressor { // ... } pub struct VariableEncoder { // Full-zip per-value compressor for variable-width data }
Import	use lance_encoding::encodings::physical::binary::BinaryMiniBlockEncoder;

I/O Contract

Direction	Type	Description
Input	DataBlock::VariableWidth	Variable-width data block with offsets and data buffers
Output	MiniBlockCompressed	Chunked mini-block data with localized offsets
Output	CompressiveEncoding	Encoding description for decompression
Output (decompress)	DataBlock::VariableWidth	Reconstructed variable-width data

Usage Examples

use lance_encoding::encodings::physical::binary::BinaryMiniBlockEncoder;
use lance_encoding::encodings::logical::primitive::miniblock::MiniBlockCompressor;

let encoder = BinaryMiniBlockEncoder::default();
let (compressed, encoding) = encoder.compress(data_block)?;

Related Pages

• Lance_format_Lance_FsstEncoding - FSST encoding applied before binary chunking
• Lance_format_Lance_ValueEncoding - Fixed-width value encoding counterpart
• Lance_format_Lance_PrimitiveEncoding - Uses binary encoding for variable-width leaves
• Lance_format_Lance_BlockCompression - General compression applied on top