Implementation:Lance format Lance BlockCompression

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

BlockCompression provides traditional block compression schemes (Zstd, LZ4) that compress entire buffers, along with the BufferCompressor trait and CompressionConfig configuration types.

Description

This module implements traditional buffer-level compression where an entire buffer is compressed into a smaller buffer. Unlike transparent encodings, block compression is opaque -- the entire buffer must be decompressed to access any value. The module defines:

CompressionConfig: Configuration holding a CompressionScheme and optional compression level.
CompressionScheme: Enum with variants None, Fsst, Zstd, and Lz4.
BufferCompressor trait: Interface for compress/decompress operations on raw byte buffers.
GeneralBufferCompressor: Factory for creating compressors from configuration. Supports Zstd (with lazy context caching for reuse across chunks) and LZ4.
BlockCompressor/BlockDecompressor implementations: Adapt buffer compression for the block encoding path, prepending an 8-byte header with the values buffer size.

The Zstd implementation lazily creates and reuses compression contexts to avoid repeated initialization overhead. The LZ4 implementation uses the lz4_flex crate.

Usage

Block compression is used when data patterns are not amenable to more specialized encodings. It is effective for large variable-length values like source code, documents, or serialized data. It can also be layered on top of other encodings via GeneralMiniBlockCompressor.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/physical/block.rs, Lines: 1-855

Signature

#[derive(Debug, Clone, Copy, PartialEq)]
pub struct CompressionConfig {
    pub(crate) scheme: CompressionScheme,
    pub(crate) level: Option<i32>,
}

#[derive(Debug, Clone, Copy, PartialEq)]
pub enum CompressionScheme {
    None,
    Fsst,
    Zstd,
    Lz4,
}

pub trait BufferCompressor: std::fmt::Debug + Send + Sync {
    fn compress(&self, input_buf: &[u8], output_buf: &mut Vec<u8>) -> Result<()>;
    fn decompress(&self, input_buf: &[u8], output_buf: &mut Vec<u8>) -> Result<()>;
    fn config(&self) -> CompressionConfig;
}

Import use lance_encoding::encodings::physical::block::{CompressionConfig, CompressionScheme, BufferCompressor};

I/O Contract

Direction	Type	Description
Input	`&[u8]`	Raw byte buffer to compress
Input	`CompressionConfig`	Scheme (Zstd/LZ4) and optional level
Output	`Vec<u8>`	Compressed byte buffer
Output (decompress)	`Vec<u8>`	Decompressed original bytes

Usage Examples

use lance_encoding::encodings::physical::block::{CompressionConfig, CompressionScheme, GeneralBufferCompressor};

let config = CompressionConfig::new(CompressionScheme::Zstd, Some(3));
let compressor = GeneralBufferCompressor::get_compressor(config)?;

let mut compressed = Vec::new();
compressor.compress(&input_data, &mut compressed)?;

let mut decompressed = Vec::new();
compressor.decompress(&compressed, &mut decompressed)?;

Related Pages

Lance_format_Lance_GeneralCompressor - Wraps mini-block compressors with block compression
Lance_format_Lance_FsstEncoding - FSST string compression (a specialized compression)
Lance_format_Lance_PrimitiveEncoding - Uses block compression in the full-zip path

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment