Implementation:Lance format Lance CompressionConfig

Knowledge Sources	Lance
Domains	Encoding, Columnar_Data
Last Updated	2026-02-08 19:33 GMT

Overview

The CompressionConfig module provides parameter types for configuring compression strategies on a per-column or per-data-type basis, including RLE thresholds, compression schemes, byte-stream-split modes, and wildcard pattern matching for column names.

Description

This module defines the user-facing configuration system for Lance compression. It allows fine-grained control over how data is compressed by specifying parameters at two levels:

CompressionParams:

The top-level configuration container with two maps:

• columns -- Maps column name patterns to field-level parameters. Supports exact matches and wildcard patterns (*_id, log_*, test_*_name, *).
• types -- Maps Arrow data type names (e.g., "Int32", "Float32") to field-level parameters.

When resolving parameters for a field, type-level parameters are applied first, then column-level parameters override them (highest priority). Exact column name matches take precedence over pattern matches.

CompressionFieldParams:

Per-field configuration with the following optional settings:

• rle_threshold (Option<f64>) -- When run_count < num_values * threshold, RLE is used. Range: 0.0 to 1.0.
• compression (Option<String>) -- General compression scheme: "lz4", "zstd", or "none".
• compression_level (Option<i32>) -- Level for schemes that support it (e.g., zstd levels).
• bss (Option<BssMode>) -- Byte-stream-split mode for floating-point data.
• minichunk_size (Option<i64>) -- Minimum chunk size threshold for encoding.

The merge method on CompressionFieldParams allows composing parameters from multiple sources, where non-None values in the source override the target.

BssMode:

Controls byte-stream-split encoding for floating-point data:

• Off -- Never use BSS
• On -- Always use BSS for floating-point data
• Auto -- Automatically decide based on data characteristics (default sensitivity: 0.5)

Usage

Use this module when:

• Configuring compression for specific columns or data types through the Lance writer API
• Passing compression parameters to the DefaultCompressionStrategy
• Implementing custom column-level compression tuning

Code Reference

Source Location	rust/lance-encoding/src/compression_config.rs
Key Structs	CompressionParams, CompressionFieldParams
Key Enum	BssMode
Import	use lance_encoding::compression_config::{CompressionParams, CompressionFieldParams, BssMode};

I/O Contract

CompressionParams Methods:

Method	Input	Output	Description
new()	--	CompressionParams	Empty configuration
get_field_params(name, data_type)	&str, &DataType	CompressionFieldParams	Resolved params (type + column merged)

CompressionFieldParams Methods:

Method	Input	Output	Description
merge(&mut self, other)	&CompressionFieldParams	--	Non-None values from other override self

Pattern Matching Rules:

Pattern	Matches	Does Not Match
*_id	user_id, product_id	identity
log_*	log_message, log_level	message_log
test_*_name	test_field_name	test_name
*	anything	--
exact_match	exact_match	anything else

Usage Examples

use lance_encoding::compression_config::{CompressionParams, CompressionFieldParams, BssMode};
use arrow_schema::DataType;

// Create compression parameters
let mut params = CompressionParams::new();

// Configure all Int32 columns to use LZ4 with RLE threshold 0.5
params.types.insert(
    "Int32".to_string(),
    CompressionFieldParams {
        rle_threshold: Some(0.5),
        compression: Some("lz4".to_string()),
        ..Default::default()
    },
);

// Override specific columns matching *_id pattern to use Zstd level 3
params.columns.insert(
    "*_id".to_string(),
    CompressionFieldParams {
        compression: Some("zstd".to_string()),
        compression_level: Some(3),
        rle_threshold: Some(0.3),
        ..Default::default()
    },
);

// Enable BSS for all Float64 columns
params.types.insert(
    "Float64".to_string(),
    CompressionFieldParams {
        bss: Some(BssMode::On),
        ..Default::default()
    },
);

// Resolve effective parameters for a specific field
let field_params = params.get_field_params("user_id", &DataType::Int32);
assert_eq!(field_params.compression, Some("zstd".to_string()));
assert_eq!(field_params.compression_level, Some(3));
assert_eq!(field_params.rle_threshold, Some(0.3));

Related Pages

• Lance_format_Lance_Compression_Traits - DefaultCompressionStrategy consumes CompressionParams to configure compression
• Lance_format_Lance_CoreEncoder - default_encoding_strategy_with_params accepts CompressionParams
• Lance_format_Lance_Statistics - Statistics like RunCount are compared against rle_threshold from these params