Implementation:Lance format Lance SubstraitCodec

Knowledge Sources	Lance
Domains	DataFusion_Integration, Query_Execution
Last Updated	2026-02-08 19:33 GMT

Overview

The SubstraitCodec module provides encoding and decoding of DataFusion expressions to and from the Substrait format, enabling serialized expression storage in Lance metadata.

Description

Substrait is a cross-language specification for data compute operations. This module uses it to serialize DataFusion Expr objects into a binary format that can be stored alongside Lance dataset metadata (e.g., for persisted filter expressions or index predicates). Key components include:

• encode_substrait -- Converts a DataFusion Expr into a Substrait ExtendedExpression protobuf message, serialized as a Vec<u8>. The schema must contain all fields referenced by the expression. Extension types (e.g., FSL, user-defined types) that cannot be represented in Substrait should be removed from the schema before encoding.

• parse_substrait -- Decodes a Substrait ExtendedExpression from bytes back into a DataFusion Expr. Handles schema differences by:
  • Removing extension types and placeholder fields from the Substrait schema.
  • Remapping field reference indices to account for removed fields.
  • Supporting recursive remapping through scalar functions, if/then expressions, cast expressions, singular/set comparisons, and field references.

• remove_extension_types -- Internal function that filters out unsupported field types (user-defined types, placeholder names) from the Substrait schema and builds an index mapping from old to new field positions.

• remap_expr_references -- Recursively updates all field reference indices in a Substrait expression tree according to the index mapping produced by remove_extension_types.

• count_fields -- Internal helper that counts the total number of leaf and struct fields in a Substrait type, correctly handling nested structs and lists for accurate index mapping.

Usage

Use this module when you need to:

• Persist filter or projection expressions as part of Lance dataset metadata
• Serialize expressions for cross-process or cross-language expression exchange
• Deserialize previously stored expressions when reading Lance datasets

Code Reference

Source Location

rust/lance-datafusion/src/substrait.rs

Signature

pub fn encode_substrait(
    expr: Expr,
    schema: Arc<ArrowSchema>,
    state: &SessionState,
) -> Result<Vec<u8>>

pub async fn parse_substrait(
    expr: &[u8],
    input_schema: Arc<ArrowSchema>,
    state: &SessionState,
) -> Result<Expr>

Import

use lance_datafusion::substrait::{encode_substrait, parse_substrait};

I/O Contract

Input	Type	Description
expr (encode)	Expr	The DataFusion logical expression to serialize
schema	Arc<ArrowSchema>	The Arrow schema containing all referenced fields
state	&SessionState	DataFusion session state for function resolution
expr (parse)	&[u8]	Serialized Substrait ExtendedExpression bytes
input_schema	Arc<ArrowSchema>	The Arrow schema to use for deserialization

Output	Type	Description
encode_substrait	Result<Vec<u8>>	Serialized Substrait protobuf bytes
parse_substrait	Result<Expr>	Deserialized DataFusion logical expression

Usage Examples

use lance_datafusion::substrait::{encode_substrait, parse_substrait};
use datafusion::prelude::*;
use std::sync::Arc;

// Encode an expression
let expr = col("price").gt(lit(100i64));
let bytes = encode_substrait(expr, schema.clone(), &session_state)?;

// Decode the expression
let decoded_expr = parse_substrait(&bytes, schema.clone(), &session_state).await?;

Related Pages

• Lance_format_Lance_FilterPlanner -- Planner that produces expressions suitable for Substrait encoding
• Lance_format_Lance_LogicalExpr -- Logical expression utilities used alongside Substrait serialization