Implementation:Lance format Lance BitpackingEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

InlineBitpacking is a physical encoding that discards unused higher-order bits from fixed-width integer values, compressing them in chunks of 1024 values with per-chunk bit widths.

Description

Bitpacking looks for unused higher-order bits in integer data and discards them. For example, if all values in a u32 column fit within 12 bits, only 12 bits are stored per value. The encoding operates in chunks of 1024 values (ELEMS_PER_CHUNK), where each chunk can have a different compressed bit width stored inline as a single element prefix.

Key implementation details:

The bitpack_chunked method uses the BitPacking trait for SIMD-optimized packing.
Per-chunk bit widths are derived from the Stat::BitWidth statistics on the data block.
The last chunk is padded to 1024 values for uniform processing.
Minimum buffer size for compression is calculated as (1024 * bit_width) / 8 bytes.
Supports all fixed-width integer types via the generic BitPacking trait.

The encoding is transparent because output retains fixed-width characteristics, enabling random access to any value.

Usage

Bitpacking is automatically selected by the compression strategy when integer data has a small value range. It is most effective for columns with values that cluster in a narrow numeric range, such as small enum values stored as integers or timestamps with limited variation.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/physical/bitpacking.rs, Lines: 1-657

Signature

#[derive(Debug, Default)]
pub struct InlineBitpacking {
    uncompressed_bit_width: u64,
}

impl InlineBitpacking {
    pub fn new(uncompressed_bit_width: u64) -> Self;
    pub fn from_description(description: &pb21::InlineBitpacking) -> Self;
    pub fn min_size_bytes(compressed_bit_width: u64) -> u64;
}

impl MiniBlockCompressor for InlineBitpacking {
    fn compress(&self, page: DataBlock) -> Result<(MiniBlockCompressed, CompressiveEncoding)>;
}

impl MiniBlockDecompressor for InlineBitpacking {
    fn decompress(&self, data: Vec<LanceBuffer>, num_values: u64) -> Result<DataBlock>;
}

Import use lance_encoding::encodings::physical::bitpacking::InlineBitpacking;

I/O Contract

Direction	Type	Description
Input	`DataBlock::FixedWidth`	Fixed-width integer data block with bit-width statistics
Output	`MiniBlockCompressed`	Bitpacked chunks of 1024 values each with inline bit widths
Output	`CompressiveEncoding`	Encoding description with original bit width
Output (decompress)	`DataBlock::FixedWidth`	Unpacked data restored to original bit width

Usage Examples

use lance_encoding::encodings::physical::bitpacking::InlineBitpacking;
use lance_encoding::encodings::logical::primitive::miniblock::MiniBlockCompressor;

let encoder = InlineBitpacking::new(32); // 32-bit original width
let (compressed, encoding) = encoder.compress(data_block)?;

Related Pages

Lance_format_Lance_ValueEncoding - Uncompressed value encoding (fallback)
Lance_format_Lance_RleEncoding - RLE encoding (alternative compression)
Lance_format_Lance_GeneralCompressor - Can wrap bitpacking with LZ4/Zstd
Lance_format_Lance_MiniBlockCompressor - Trait implemented by bitpacking

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment