Implementation:Lance format Lance MiniBlockCompressor

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

MiniBlockCompressor is the trait defining the miniblock structural encoding path in Lance 2.1, where data is compressed into small independently-decodable chunks within a single buffer.

Description

Miniblock encoding is one of the two structural encoding strategies in Lance 2.1 (the other being full-zip). Data is compressed into a series of chunks stored in a single buffer. Each chunk is encoded and decoded as a unit, enabling efficient random access at chunk granularity.

The module defines three key types:

• MiniBlockCompressed: Holds the compressed data buffers, chunk descriptors, and total value count. Multiple buffers are supported (e.g., values + validity for FSL data).
• MiniBlockChunk: Describes a single chunk with buffer_sizes: Vec<u32> (byte size of each buffer in the chunk) and log_num_values: u8 (log2 of the number of values, or 0 for the last chunk). Chunks are limited to 4096 values and approximately 8 KiB of compressed data.
• MiniBlockCompressor trait: Requires implementing compress which takes a DataBlock (a full page of data) and returns MiniBlockCompressed plus a CompressiveEncoding description.

Constants:

• MAX_MINIBLOCK_BYTES = 8186 (8 KiB minus 6 bytes of overhead)
• MAX_MINIBLOCK_VALUES = 4096

Usage

Miniblock is the default structural encoding for most data types in Lance 2.1. Implementors include ValueEncoder, InlineBitpacking, RleEncoder, BinaryMiniBlockEncoder, ByteStreamSplitEncoder, FsstMiniBlockEncoder, PackedStructFixedWidthMiniBlockEncoder, and GeneralMiniBlockCompressor.

Code Reference

Source Location	Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/logical/primitive/miniblock.rs, Lines: 1-89
Signature	pub const MAX_MINIBLOCK_BYTES: u64 = 8 * 1024 - 6; pub const MAX_MINIBLOCK_VALUES: u64 = 4096; #[derive(Debug)] pub struct MiniBlockCompressed { pub data: Vec<LanceBuffer>, pub chunks: Vec<MiniBlockChunk>, pub num_values: u64, } #[derive(Debug)] pub struct MiniBlockChunk { pub buffer_sizes: Vec<u32>, pub log_num_values: u8, } impl MiniBlockChunk { pub fn num_values(&self, vals_in_prev_blocks: u64, total_num_values: u64) -> u64; } pub trait MiniBlockCompressor: std::fmt::Debug + Send + Sync { fn compress(&self, page: DataBlock) -> Result<(MiniBlockCompressed, CompressiveEncoding)>; }
Import	use lance_encoding::encodings::logical::primitive::miniblock::{MiniBlockCompressor, MiniBlockCompressed, MiniBlockChunk};

I/O Contract

Direction	Type	Description
Input	DataBlock	Full page of data to compress into mini-block chunks
Output	MiniBlockCompressed	Compressed data with chunk descriptors
Output	CompressiveEncoding	Encoding description for decompression

Usage Examples

use lance_encoding::encodings::logical::primitive::miniblock::{
    MiniBlockCompressor, MiniBlockCompressed, MiniBlockChunk,
};
use lance_encoding::encodings::physical::value::ValueEncoder;

// Any MiniBlockCompressor implementor can be used:
let compressor: Box<dyn MiniBlockCompressor> = Box::new(ValueEncoder::default());
let (compressed, encoding) = compressor.compress(data_block)?;

// Inspect chunks:
for (i, chunk) in compressed.chunks.iter().enumerate() {
    let num_vals = chunk.num_values(prev_vals, compressed.num_values);
    // Each chunk can be decoded independently
}

Related Pages

• Lance_format_Lance_FullZipCompressor - Alternative structural encoding (full-zip)
• Lance_format_Lance_PrimitiveEncoding - Orchestrates miniblock encoding
• Lance_format_Lance_ValueEncoding - Default MiniBlockCompressor implementation
• Lance_format_Lance_BitpackingEncoding - Bitpacking MiniBlockCompressor
• Lance_format_Lance_RleEncoding - RLE MiniBlockCompressor
• Lance_format_Lance_BinaryEncoding - Binary MiniBlockCompressor
• Lance_format_Lance_GeneralCompressor - Compression wrapper for MiniBlockCompressor