Implementation:Lance format Lance DictEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

DictEncoding provides dictionary encoding utilities for Lance, including null normalization, dictionary construction for both fixed-width and variable-width data, and cardinality-based encoding decisions.

Description

This module implements dictionary encoding for Lance columns. Dictionary encoding replaces repeated values with integer indices into a lookup table (dictionary). The key components include:

normalize_dict_nulls: Normalizes Arrow dictionary arrays so that all nulls are in the keys (not the values). This ensures nulls can be stored as rep-def values consistently.
dict_encode_variable_width: Builds a dictionary from variable-width data (strings, binary) using a hash map. Returns the dictionary and indices as separate data blocks, or None if the cardinality exceeds the configured ratio.
dict_encode_fixed_width: Builds a dictionary from fixed-width data. Supports all fixed-width primitive types.

Dictionary indices use 32-bit integers (DICT_INDICES_BITS_PER_VALUE = 32). The encoding is selected when the cardinality ratio (unique values / total values) is below a configurable threshold set via the DICT_SIZE_RATIO_META_KEY field metadata.

Usage

Dictionary encoding is automatically applied by the compression strategy when columns have sufficient value repetition. It is effective for categorical data, enums, and string columns with limited unique values.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/logical/primitive/dict.rs, Lines: 1-517

Signature

pub const DICT_FIXED_WIDTH_BITS_PER_VALUE: u64 = 128;
pub const DICT_INDICES_BITS_PER_VALUE: u64 = 32;

pub fn normalize_dict_nulls(array: Arc<dyn Array>) -> Result<Arc<dyn Array>>;

fn dict_encode_variable_width<T>(
    variable_width_data_block: &VariableWidthBlock,
    bits_per_offset: u8,
    cardinality: u64,
) -> Option<(DataBlock, DataBlock)>;

Import use lance_encoding::encodings::logical::primitive::dict::{normalize_dict_nulls, DICT_INDICES_BITS_PER_VALUE};

I/O Contract

Direction	Type	Description
Input	`Arc<dyn Array>` (Dictionary)	Arrow dictionary array with potential value-nulls
Input	`VariableWidthBlock` or `FixedWidthDataBlock`	Raw data block for dictionary construction
Output	Normalized `DictionaryArray`	Dictionary with nulls moved to keys
Output	`(DataBlock, DataBlock)`	Dictionary data block and indices data block

Usage Examples

use lance_encoding::encodings::logical::primitive::dict::normalize_dict_nulls;
use std::sync::Arc;

// Normalize nulls in a dictionary array before encoding
let normalized = normalize_dict_nulls(dict_array)?;

Related Pages

Lance_format_Lance_PrimitiveEncoding - Uses dictionary encoding as a compression option
Lance_format_Lance_BinaryEncoding - Binary encoding for dictionary values
Lance_format_Lance_ValueEncoding - Value encoding for dictionary indices

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment