Implementation:Lance format Lance FsstEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

FsstMiniBlockEncoder is a physical encoding that applies Fast Static Symbol Table (FSST) compression to variable-width string and binary data, then chunks the result using binary mini-block encoding.

Description

FSST is a lightweight string compression algorithm that builds a small symbol table mapping frequent byte sequences to single-byte symbols. This module provides adapters for both the miniblock and per-value (full-zip) encoding paths.

The encoding pipeline works in two stages:

FSST compression: The FsstCompressed::fsst_compress method compresses variable-width data using the FSST algorithm, producing compressed data with a symbol table. Supports both 32-bit and 64-bit offset types.
Binary chunking: The compressed data is then processed by BinaryMiniBlockEncoder to produce mini-block chunks.

The symbol table is stored per disk page in the encoding description. The per-value variant (FsstPerValueEncoder) compresses each value independently for the full-zip path.

The encoding is transparent since individual values can be decompressed independently (in the per-value variant) or within their mini-block chunk.

Usage

FSST encoding is selected by the compression strategy for string and binary columns when FSST compression is expected to yield good ratios. It is effective for columns with repetitive string patterns such as URLs, file paths, or structured text.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/physical/fsst.rs, Lines: 1-411

Signature

#[derive(Debug, Default)]
pub struct FsstMiniBlockEncoder {
    minichunk_size: Option<i64>,
}

impl FsstMiniBlockEncoder {
    pub fn new(minichunk_size: Option<i64>) -> Self;
}

impl MiniBlockCompressor for FsstMiniBlockEncoder {
    fn compress(&self, data: DataBlock) -> Result<(MiniBlockCompressed, CompressiveEncoding)>;
}

Import use lance_encoding::encodings::physical::fsst::FsstMiniBlockEncoder;

I/O Contract

Direction	Type	Description
Input	`DataBlock::VariableWidth`	Variable-width string or binary data
Output	`MiniBlockCompressed`	FSST-compressed data in binary mini-block chunks
Output	`CompressiveEncoding`	Encoding with FSST symbol table and inner binary encoding
Output (decompress)	`DataBlock::VariableWidth`	Decompressed original string/binary data

Usage Examples

use lance_encoding::encodings::physical::fsst::FsstMiniBlockEncoder;
use lance_encoding::encodings::logical::primitive::miniblock::MiniBlockCompressor;

let encoder = FsstMiniBlockEncoder::default();
let (compressed, encoding) = encoder.compress(string_data_block)?;

Related Pages

Lance_format_Lance_BinaryEncoding - Binary encoding used after FSST compression
Lance_format_Lance_BlockCompression - Alternative general compression
Lance_format_Lance_DictEncoding - Dictionary encoding (alternative for repetitive strings)
Lance_format_Lance_GeneralCompressor - Can layer LZ4/Zstd on top of FSST

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment