Implementation:Lance format Lance MapEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

MapStructuralEncoder is a logical encoding for map fields, treating them as List<Struct<key, value>> and encoding offsets into rep/def levels while delegating entries to a child struct encoder.

Description

In Arrow, a Map is physically represented as List<Struct<key, value>>. The map encoding leverages this by adding the map offsets and validity to the RepDefBuilder, then converting the MapArray into a ListArray to use the ListArrayExt trait for garbage filtering and value trimming. The resulting entries (struct array of key-value pairs) are passed to the child encoder. The scheduler (StructuralMapScheduler) delegates to its child, and the decoder (StructuralMapDecoder) reconstructs the map array by building a MapArray from the decoded struct entries and offsets extracted from rep/def levels.

Usage

Use this encoding for map-typed columns. It is automatically selected by the encoding framework when a Map field is encountered.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/logical/map.rs, Lines: 1-728

Signature

pub struct MapStructuralEncoder {
    keep_original_array: bool,
    child: Box<dyn FieldEncoder>,
}

impl MapStructuralEncoder {
    pub fn new(keep_original_array: bool, child: Box<dyn FieldEncoder>) -> Self;
}

pub struct StructuralMapScheduler {
    child: Box<dyn StructuralFieldScheduler>,
}

pub struct StructuralMapDecoder {
    child: Box<dyn StructuralFieldDecoder>,
    data_type: DataType,
}

Import use lance_encoding::encodings::logical::map::{MapStructuralEncoder, StructuralMapScheduler, StructuralMapDecoder};

I/O Contract

Direction	Type	Description
Input	`ArrayRef` (Map)	Map array with key-value entries
Input	`RepDefBuilder`	Builder that receives map offsets and validity
Output	`Vec<EncodeTask>`	Encode tasks for the entries struct array
Output (decode)	`DecodedArray`	Reconstructed Map array with offsets and entries

Usage Examples

use lance_encoding::encodings::logical::map::MapStructuralEncoder;

// Created by the encoding framework for Map<K, V> columns
let encoder = MapStructuralEncoder::new(
    false,
    child_struct_encoder,
);

Related Pages

Lance_format_Lance_ListEncoding - List encoding (maps share the list structure)
Lance_format_Lance_StructEncoding - Struct encoding for map entries
Lance_format_Lance_PrimitiveEncoding - Leaf value encoding

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment