Implementation:Lance format Lance Compression Traits

Knowledge Sources	Lance
Domains	Encoding, Columnar_Data
Last Updated	2026-02-08 19:33 GMT

Overview

The Compression Traits module defines the strategy interfaces and default implementation for selecting and applying compression algorithms (RLE, bitpacking, FSST, byte-stream-split, general block compression) to Lance 2.1+ encoded data.

Description

This module is the entry point for the compression step that occurs after structural encoding in Lance 2.1+. The encoding pipeline first shreds input into leaf arrays and handles validity/offsets, then a structural encoding (miniblock or fullzip) is chosen, and finally the data is compressed using the strategies defined here.

Core Traits:

• BlockCompressor -- Compresses an entire DataBlock into a single opaque LanceBuffer. This is the most general but least random-access-friendly form. Used only for metadata buffers (e.g., dictionaries, rep/def miniblock chunks).
• CompressionStrategy -- A trait to select which compression to apply for a given field and data block. It can create block compressors, per-value compressors, or miniblock compressors.
• DecompressionStrategy -- The read-side counterpart that creates decompressors from encoded metadata (protobuf CompressiveEncoding descriptors).

Default Implementation (DefaultCompressionStrategy):

The default strategy examines data statistics and field metadata to choose the best compression:

1. Byte-stream-split (BSS) for floating-point data when it improves compressibility
2. RLE when run count is below a threshold relative to the total value count
3. Bitpacking (inline or out-of-line) when bit widths allow meaningful reduction
4. FSST for variable-width string data meeting minimum size requirements
5. General block compression (LZ4/Zstd) wrapping other compressors for large blocks (>32KB)

User-configurable parameters are pulled from field metadata keys (lance-encoding:compression, lance-encoding:compression-level, lance-encoding:rle-threshold, lance-encoding:bss) and from CompressionParams.

Usage

Use this module when:

• Implementing a new compression algorithm for Lance encoding
• Configuring per-column or per-type compression through CompressionParams
• Decoding compressed data by using the DecompressionStrategy to reconstruct decompressors from stored encoding metadata

Code Reference

Source Location	rust/lance-encoding/src/compression.rs
Key Traits	BlockCompressor, CompressionStrategy, DecompressionStrategy
Default Impl	DefaultCompressionStrategy, DefaultDecompressionStrategy
Import	use lance_encoding::compression::{CompressionStrategy, DefaultCompressionStrategy};

I/O Contract

CompressionStrategy Trait Methods:

Method	Input	Output
create_block_compressor	&Field, &DataBlock	Result<(Box<dyn BlockCompressor>, CompressiveEncoding)>
create_per_value	&Field, &DataBlock	Result<Box<dyn PerValueCompressor>>
create_miniblock_compressor	&Field, &DataBlock	Result<Box<dyn MiniBlockCompressor>>

BlockCompressor Trait:

Method	Input	Output
compress	DataBlock	Result<LanceBuffer>

Compression Selection Heuristics:

Algorithm	Trigger Condition
RLE	run_count < num_values * threshold (default threshold: 0.5)
Bitpacking	Data is 8/16/32/64-bit fixed width with bit widths smaller than type width
BSS	Floating-point data where byte-stream-split improves compressibility
FSST	Variable-width data with max_length >= 12 and data_size >= minimum
General (LZ4/Zstd)	Block size > 32KB, or user-specified compression scheme

Usage Examples

use lance_encoding::compression::{CompressionStrategy, DefaultCompressionStrategy};
use lance_encoding::compression_config::CompressionParams;
use lance_core::datatypes::Field;

// Create with default parameters
let strategy = DefaultCompressionStrategy::new();

// Create with user-configured parameters
let params = CompressionParams::new();
let strategy = DefaultCompressionStrategy::with_params(params);

// The strategy is used internally by the encoding pipeline.
// Given a field and data block, it selects the appropriate compressor:
// let compressor = strategy.create_miniblock_compressor(&field, &data_block)?;

Related Pages

• Lance_format_Lance_CompressionConfig - Provides the CompressionParams and CompressionFieldParams types consumed by this module
• Lance_format_Lance_DataBlock - The DataBlock enum that compression operates on
• Lance_format_Lance_LanceBuffer - The buffer type produced by compression
• Lance_format_Lance_Statistics - Statistics (Stat::RunCount, Stat::BitWidth) that drive compression selection
• Lance_format_Lance_EncodingFormat - Protobuf utilities that describe compression in file metadata
• Lance_format_Lance_CoreEncoder - The encoder pipeline that calls into compression strategies