Implementation:Lance format Lance BitpackingEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

InlineBitpacking is a physical encoding that discards unused higher-order bits from fixed-width integer values, compressing them in chunks of 1024 values with per-chunk bit widths.

Description

Bitpacking looks for unused higher-order bits in integer data and discards them. For example, if all values in a u32 column fit within 12 bits, only 12 bits are stored per value. The encoding operates in chunks of 1024 values (ELEMS_PER_CHUNK), where each chunk can have a different compressed bit width stored inline as a single element prefix.

Key implementation details:

• The bitpack_chunked method uses the BitPacking trait for SIMD-optimized packing.
• Per-chunk bit widths are derived from the Stat::BitWidth statistics on the data block.
• The last chunk is padded to 1024 values for uniform processing.
• Minimum buffer size for compression is calculated as (1024 * bit_width) / 8 bytes.
• Supports all fixed-width integer types via the generic BitPacking trait.

The encoding is transparent because output retains fixed-width characteristics, enabling random access to any value.

Usage

Bitpacking is automatically selected by the compression strategy when integer data has a small value range. It is most effective for columns with values that cluster in a narrow numeric range, such as small enum values stored as integers or timestamps with limited variation.

Code Reference

Source Location	Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/physical/bitpacking.rs, Lines: 1-657
Signature	#[derive(Debug, Default)] pub struct InlineBitpacking { uncompressed_bit_width: u64, } impl InlineBitpacking { pub fn new(uncompressed_bit_width: u64) -> Self; pub fn from_description(description: &pb21::InlineBitpacking) -> Self; pub fn min_size_bytes(compressed_bit_width: u64) -> u64; } impl MiniBlockCompressor for InlineBitpacking { fn compress(&self, page: DataBlock) -> Result<(MiniBlockCompressed, CompressiveEncoding)>; } impl MiniBlockDecompressor for InlineBitpacking { fn decompress(&self, data: Vec<LanceBuffer>, num_values: u64) -> Result<DataBlock>; }
Import	use lance_encoding::encodings::physical::bitpacking::InlineBitpacking;

I/O Contract

Direction	Type	Description
Input	DataBlock::FixedWidth	Fixed-width integer data block with bit-width statistics
Output	MiniBlockCompressed	Bitpacked chunks of 1024 values each with inline bit widths
Output	CompressiveEncoding	Encoding description with original bit width
Output (decompress)	DataBlock::FixedWidth	Unpacked data restored to original bit width

Usage Examples

use lance_encoding::encodings::physical::bitpacking::InlineBitpacking;
use lance_encoding::encodings::logical::primitive::miniblock::MiniBlockCompressor;

let encoder = InlineBitpacking::new(32); // 32-bit original width
let (compressed, encoding) = encoder.compress(data_block)?;

Related Pages

• Implementation:Lance_format_Lance_ValueEncoding - Uncompressed value encoding (fallback)
• Implementation:Lance_format_Lance_RleEncoding - RLE encoding (alternative compression)
• Implementation:Lance_format_Lance_GeneralCompressor - Can wrap bitpacking with LZ4/Zstd
• Implementation:Lance_format_Lance_MiniBlockCompressor - Trait implemented by bitpacking