Implementation:Lance format Lance ListEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

ListStructuralEncoder is a logical encoding for variable-size list fields that encodes list offsets into repetition/definition levels and delegates the flattened values to a child encoder.

Description

The list encoding handles both List<i32> and LargeList<i64> Arrow types. During encoding, the list offsets and validity are added to the RepDefBuilder, which tracks the list structure. Garbage values (values under null list entries with non-zero offsets) are filtered, and the values array is trimmed to only include actually used entries. The flattened child values are then passed to the child encoder. The scheduler (StructuralListScheduler) delegates scheduling directly to the child, and the decoder (StructuralListDecoder) rebuilds list arrays from the rep/def levels by extracting offsets from the unraveled rep/def data.

Usage

Use this encoding for any variable-length list column (List or LargeList). It is automatically selected by the encoding framework when list-typed fields are encountered.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/logical/list.rs, Lines: 1-892

Signature

pub struct ListStructuralEncoder {
    keep_original_array: bool,
    child: Box<dyn FieldEncoder>,
}

impl ListStructuralEncoder {
    pub fn new(keep_original_array: bool, child: Box<dyn FieldEncoder>) -> Self;
}

pub struct StructuralListScheduler {
    child: Box<dyn StructuralFieldScheduler>,
}

impl StructuralListScheduler {
    pub fn new(child: Box<dyn StructuralFieldScheduler>) -> Self;
}

Import use lance_encoding::encodings::logical::list::{ListStructuralEncoder, StructuralListScheduler};

I/O Contract

Direction	Type	Description
Input	`ArrayRef` (List or LargeList)	Variable-size list array
Input	`RepDefBuilder`	Builder that receives list offsets and validity
Output	`Vec<EncodeTask>`	Encode tasks for the flattened child values
Output (decode)	`DecodedArray`	Reconstructed List or LargeList array

Usage Examples

use lance_encoding::encodings::logical::list::ListStructuralEncoder;

// Created by the encoding framework for List<T> columns
let encoder = ListStructuralEncoder::new(
    false, // deep copy nulls for safety
    child_encoder,
);

Related Pages

Lance_format_Lance_FixedSizeListEncoding - Fixed-size list encoding
Lance_format_Lance_MapEncoding - Map encoding (uses list internally)
Lance_format_Lance_PrimitiveEncoding - Primitive encoder for leaf values

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment