Implementation:Lance format Lance RleEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

RleEncoder is a physical encoding that compresses data with repeated consecutive values using run-length encoding, storing unique values and their repeat counts in dual buffers.

Description

RLE (Run-Length Encoding) stores repeated consecutive values as (value, length) pairs. The encoding uses a dual-buffer format:

• Values buffer: Stores unique values in their original data type
• Lengths buffer: Stores the repeat count for each value as u8

When a run exceeds 255 values (the maximum for a u8 length), it is split into multiple runs of 255 followed by a final run with the remainder.

Key implementation details:

• Supports all fixed-width types: 8-bit, 16-bit, 32-bit, and 64-bit integers and floats.
• Uses a rolling chunk algorithm that processes values in power-of-2 chunks (up to 2048 values per chunk).
• Each chunk's buffer_sizes in the mini-block format identifies its slice within the global values and lengths buffers.
• RLE is selected when the run count is less than 50% of total values, indicating sufficient repetition.
• Also implements BlockCompressor for the block path, using format: [8-byte header: values buffer size][values buffer][lengths buffer].

Usage

RLE is automatically selected by the compression strategy for columns with high value repetition, such as status codes, category identifiers, boolean flags, or sorted columns.

Code Reference

Source Location	Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/physical/rle.rs, Lines: 1-1255
Signature	#[derive(Debug, Default)] pub struct RleEncoder; impl RleEncoder { pub fn new() -> Self; } impl MiniBlockCompressor for RleEncoder { fn compress(&self, page: DataBlock) -> Result<(MiniBlockCompressed, CompressiveEncoding)>; } impl MiniBlockDecompressor for RleEncoder { fn decompress(&self, data: Vec<LanceBuffer>, num_values: u64) -> Result<DataBlock>; } impl BlockCompressor for RleEncoder { fn compress(&self, data: DataBlock) -> Result<(DataBlock, CompressiveEncoding)>; }
Import	use lance_encoding::encodings::physical::rle::RleEncoder;

I/O Contract

Direction	Type	Description
Input	DataBlock::FixedWidth	Fixed-width data block with consecutive repeated values
Output	MiniBlockCompressed	Dual-buffer (values + lengths) in mini-block chunks
Output	CompressiveEncoding	RLE encoding description with original bit width
Output (decompress)	DataBlock::FixedWidth	Expanded data with runs restored to individual values

Usage Examples

use lance_encoding::encodings::physical::rle::RleEncoder;
use lance_encoding::encodings::logical::primitive::miniblock::MiniBlockCompressor;

let encoder = RleEncoder::new();
let (compressed, encoding) = encoder.compress(data_block)?;
// compressed.data contains [values_buffer, lengths_buffer]

Related Pages

• Lance_format_Lance_BitpackingEncoding - Alternative compression for small-range values
• Lance_format_Lance_ValueEncoding - Uncompressed baseline encoding
• Lance_format_Lance_GeneralCompressor - Can wrap RLE with LZ4/Zstd
• Lance_format_Lance_MiniBlockCompressor - Trait implemented by RLE