Implementation:Lance format Lance DictEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

DictEncoding provides dictionary encoding utilities for Lance, including null normalization, dictionary construction for both fixed-width and variable-width data, and cardinality-based encoding decisions.

Description

This module implements dictionary encoding for Lance columns. Dictionary encoding replaces repeated values with integer indices into a lookup table (dictionary). The key components include:

• normalize_dict_nulls: Normalizes Arrow dictionary arrays so that all nulls are in the keys (not the values). This ensures nulls can be stored as rep-def values consistently.
• dict_encode_variable_width: Builds a dictionary from variable-width data (strings, binary) using a hash map. Returns the dictionary and indices as separate data blocks, or None if the cardinality exceeds the configured ratio.
• dict_encode_fixed_width: Builds a dictionary from fixed-width data. Supports all fixed-width primitive types.

Dictionary indices use 32-bit integers (DICT_INDICES_BITS_PER_VALUE = 32). The encoding is selected when the cardinality ratio (unique values / total values) is below a configurable threshold set via the DICT_SIZE_RATIO_META_KEY field metadata.

Usage

Dictionary encoding is automatically applied by the compression strategy when columns have sufficient value repetition. It is effective for categorical data, enums, and string columns with limited unique values.

Code Reference

Source Location	Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/logical/primitive/dict.rs, Lines: 1-517
Signature	pub const DICT_FIXED_WIDTH_BITS_PER_VALUE: u64 = 128; pub const DICT_INDICES_BITS_PER_VALUE: u64 = 32; pub fn normalize_dict_nulls(array: Arc<dyn Array>) -> Result<Arc<dyn Array>>; fn dict_encode_variable_width<T>( variable_width_data_block: &VariableWidthBlock, bits_per_offset: u8, cardinality: u64, ) -> Option<(DataBlock, DataBlock)>;
Import	use lance_encoding::encodings::logical::primitive::dict::{normalize_dict_nulls, DICT_INDICES_BITS_PER_VALUE};

I/O Contract

Direction	Type	Description
Input	Arc<dyn Array> (Dictionary)	Arrow dictionary array with potential value-nulls
Input	VariableWidthBlock or FixedWidthDataBlock	Raw data block for dictionary construction
Output	Normalized DictionaryArray	Dictionary with nulls moved to keys
Output	(DataBlock, DataBlock)	Dictionary data block and indices data block

Usage Examples

use lance_encoding::encodings::logical::primitive::dict::normalize_dict_nulls;
use std::sync::Arc;

// Normalize nulls in a dictionary array before encoding
let normalized = normalize_dict_nulls(dict_array)?;

Related Pages

• Implementation:Lance_format_Lance_PrimitiveEncoding - Uses dictionary encoding as a compression option
• Implementation:Lance_format_Lance_BinaryEncoding - Binary encoding for dictionary values
• Implementation:Lance_format_Lance_ValueEncoding - Value encoding for dictionary indices