Implementation:Lance format Lance ByteStreamSplit

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

ByteStreamSplitEncoder is a physical encoding that reorganizes multi-byte floating-point values by byte position to improve downstream compression ratios.

Description

Byte Stream Split (BSS) is a data transformation technique that splits multi-byte values by byte position, creating separate streams for each byte position across all values. For example, given four f32 values, BSS collects all first bytes together, all second bytes together, and so on. This transformation is not compression itself but dramatically improves the effectiveness of downstream general-purpose compression (LZ4, Zstd) because:

High-order bytes of timestamps often share values
Floating-point exponent bytes cluster around common values
Sensor data often varies in a small range, sharing many byte positions

The encoder supports:

f32 (32-bit): Chunks of up to 1024 values (4 KiB per chunk)
f64 (64-bit): Chunks of up to 512 values (4 KiB per chunk)

All chunks share a single global buffer. The decompressor reverses the byte transposition to reconstruct original values.

Usage

BSS is selected by the compression strategy for floating-point columns when the BssMode configuration enables it. It is typically combined with general-purpose compression for maximum effect.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/physical/byte_stream_split.rs, Lines: 1-442

Signature

#[derive(Debug, Clone)]
pub struct ByteStreamSplitEncoder {
    bits_per_value: usize,
}

impl ByteStreamSplitEncoder {
    pub fn new(bits_per_value: usize) -> Self;
}

impl MiniBlockCompressor for ByteStreamSplitEncoder {
    fn compress(&self, page: DataBlock) -> Result<(MiniBlockCompressed, CompressiveEncoding)>;
}

Import use lance_encoding::encodings::physical::byte_stream_split::ByteStreamSplitEncoder;

I/O Contract

Direction	Type	Description
Input	`DataBlock::FixedWidth`	Fixed-width floating-point data (f32 or f64)
Output	`MiniBlockCompressed`	Byte-transposed data in mini-block chunks
Output	`CompressiveEncoding`	BSS encoding description wrapping a flat inner encoding
Output (decompress)	`DataBlock::FixedWidth`	Original floating-point data restored from byte streams

Usage Examples

use lance_encoding::encodings::physical::byte_stream_split::ByteStreamSplitEncoder;
use lance_encoding::encodings::logical::primitive::miniblock::MiniBlockCompressor;

// For f32 columns
let encoder = ByteStreamSplitEncoder::new(32);
let (compressed, encoding) = encoder.compress(float_data_block)?;

// For f64 columns
let encoder = ByteStreamSplitEncoder::new(64);
let (compressed, encoding) = encoder.compress(double_data_block)?;

Related Pages

Lance_format_Lance_ValueEncoding - Base value encoding without transformation
Lance_format_Lance_GeneralCompressor - Often combined with BSS for final compression
Lance_format_Lance_MiniBlockCompressor - Trait implemented by BSS encoder

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment