Implementation:Lance format Lance GeneralCompressor

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

GeneralMiniBlockCompressor wraps another mini-block compressor and applies general-purpose compression (LZ4 or Zstd) to the resulting buffers for additional size reduction.

Description

This module provides a composable compression wrapper that applies traditional block compression on top of any existing mini-block compressor. The workflow is:

The inner compressor encodes the data into mini-block chunks.
Each chunk's first buffer is independently compressed using the configured scheme (LZ4 or Zstd).
If compression is not effective (compressed size >= original size), the original uncompressed data is returned.
The encoding description wraps the inner encoding with compression metadata.

A minimum buffer size threshold of 4 KiB (MIN_BUFFER_SIZE_FOR_COMPRESSION) prevents overhead on small buffers. The corresponding GeneralMiniBlockDecompressor reverses the process: it decompresses the first buffer of each chunk, then delegates to the inner decompressor.

Usage

The general compressor is used when the compression strategy determines that layering LZ4 or Zstd on top of another encoding (such as bitpacking, RLE, or value encoding) would further reduce size. It is selected automatically based on compression configuration.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/physical/general.rs, Lines: 1-663

Signature

#[derive(Debug)]
pub struct GeneralMiniBlockCompressor {
    inner: Box<dyn MiniBlockCompressor>,
    compression: CompressionConfig,
}

impl GeneralMiniBlockCompressor {
    pub fn new(inner: Box<dyn MiniBlockCompressor>, compression: CompressionConfig) -> Self;
}

impl MiniBlockCompressor for GeneralMiniBlockCompressor {
    fn compress(&self, page: DataBlock) -> Result<(MiniBlockCompressed, CompressiveEncoding)>;
}

#[derive(Debug)]
pub struct GeneralMiniBlockDecompressor {
    inner: Box<dyn MiniBlockDecompressor>,
    compression: CompressionConfig,
}

impl GeneralMiniBlockDecompressor {
    pub fn new(inner: Box<dyn MiniBlockDecompressor>, compression: CompressionConfig) -> Self;
}

Import use lance_encoding::encodings::physical::general::{GeneralMiniBlockCompressor, GeneralMiniBlockDecompressor};

I/O Contract

Direction	Type	Description
Input	`DataBlock`	Data to compress (passed through inner compressor first)
Input	`CompressionConfig`	LZ4 or Zstd configuration with optional level
Output	`MiniBlockCompressed`	Mini-block data with compressed first buffers
Output	`CompressiveEncoding`	Wrapped encoding description with compression metadata
Output (decompress)	`DataBlock`	Fully decompressed data from both layers

Usage Examples

use lance_encoding::encodings::physical::general::GeneralMiniBlockCompressor;
use lance_encoding::encodings::physical::block::{CompressionConfig, CompressionScheme};
use lance_encoding::encodings::physical::value::ValueEncoder;

let inner = Box::new(ValueEncoder::default());
let config = CompressionConfig::new(CompressionScheme::Lz4, Some(0));
let compressor = GeneralMiniBlockCompressor::new(inner, config);

let (compressed, encoding) = compressor.compress(data_block)?;

Related Pages

Lance_format_Lance_BlockCompression - Underlying buffer compression implementations
Lance_format_Lance_ValueEncoding - Common inner compressor
Lance_format_Lance_BitpackingEncoding - Another common inner compressor
Lance_format_Lance_RleEncoding - Another common inner compressor
Lance_format_Lance_MiniBlockCompressor - Trait implemented and wrapped by this module

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment