Implementation:Lance format Lance FilterPlanner

Knowledge Sources	Lance
Domains	DataFusion_Integration, Query_Execution
Last Updated	2026-02-08 19:33 GMT

Overview

The FilterPlanner module provides Lance's SQL expression parser and planner, converting SQL strings into DataFusion logical and physical expressions with Lance-specific extensions.

Description

This module is the primary entry point for parsing SQL-like filter and projection expressions within Lance. Key components include:

• Planner -- The main struct that wraps a schema and a LanceContextProvider. It provides methods for parsing, optimizing, and converting expressions:
  • new(schema) -- Creates a new planner for the given Arrow schema.
  • with_enable_relations(bool) -- Enables relation-qualified column references (e.g., table.column).
  • parse_filter(filter) -- Parses a SQL WHERE clause string into a DataFusion Expr.
  • parse_expr(expr) -- Parses a general SQL expression string into a DataFusion Expr.
  • optimize_expr(expr) -- Simplifies an expression using DataFusion's expression simplifier.
  • create_physical_expr(expr) -- Converts a logical Expr into a physical PhysicalExpr.
  • column_names_in_expr(expr) -- Extracts all column names referenced in an expression tree.
• LanceContextProvider -- An internal adapter implementing DataFusion's ContextProvider trait. It provides access to Lance's registered UDFs, aggregate functions, and window functions through a cached session context.
• CastListF16Udf -- A custom scalar UDF (_cast_list_f16) that casts FixedSizeList<Float32> or List<Float32> to their Float16 equivalents, used for half-precision vector operations.

The planner handles Lance-specific SQL extensions including:

• Nested struct field access via dot notation (e.g., metadata.name)
• Array literal parsing (e.g., [1.0, 2.0, 3.0] as fixed-size lists)
• Half-precision float casting for vector columns
• Case-insensitive column name resolution
• Type-safe literal coercion via the safe_coerce_scalar pipeline

Usage

Use the Planner when you need to:

• Parse user-provided SQL filter strings for dataset scanning
• Convert SQL projection expressions to DataFusion expressions
• Build physical expressions for filter pushdown
• Extract column dependencies from filter expressions

Code Reference

Source Location

rust/lance-datafusion/src/planner.rs

Signature

pub struct Planner {
    schema: SchemaRef,
    context_provider: LanceContextProvider,
    enable_relations: bool,
}

impl Planner {
    pub fn new(schema: SchemaRef) -> Self;
    pub fn with_enable_relations(self, enable_relations: bool) -> Self;
    pub fn parse_filter(&self, filter: &str) -> Result<Expr>;
    pub fn parse_expr(&self, expr: &str) -> Result<Expr>;
    pub fn optimize_expr(&self, expr: Expr) -> Result<Expr>;
    pub fn create_physical_expr(&self, expr: &Expr) -> Result<Arc<dyn PhysicalExpr>>;
    pub fn column_names_in_expr(expr: &Expr) -> Vec<String>;
}

Import

use lance_datafusion::planner::Planner;

I/O Contract

Input	Type	Description
schema	SchemaRef	Arrow schema for column resolution and type checking
filter / expr	&str	SQL string to parse as a filter predicate or general expression

Output	Type	Description
parse_filter	Result<Expr>	A DataFusion logical expression representing the parsed filter
parse_expr	Result<Expr>	A DataFusion logical expression representing the parsed expression
create_physical_expr	Result<Arc<dyn PhysicalExpr>>	A physical expression ready for evaluation
column_names_in_expr	Vec<String>	List of all column names referenced in the expression

Usage Examples

use lance_datafusion::planner::Planner;
use arrow_schema::{Schema, Field, DataType};
use std::sync::Arc;

let schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new("name", DataType::Utf8, true),
    Field::new("score", DataType::Float32, false),
]));

let planner = Planner::new(schema);

// Parse a filter expression
let filter = planner.parse_filter("id > 10 AND name IS NOT NULL")?;

// Parse a projection expression
let expr = planner.parse_expr("score * 2.0")?;

// Get column names referenced by the filter
let columns = Planner::column_names_in_expr(&filter);
// columns: ["id", "name"]

// Convert to a physical expression
let physical = planner.create_physical_expr(&filter)?;

Related Pages

• Lance_format_Lance_LogicalExpr -- Logical expression resolution called after parsing
• Lance_format_Lance_SafeScalarValue -- Scalar coercion used during expression resolution
• Lance_format_Lance_ProjectionPlan -- Projection planning that uses the Planner
• Lance_format_Lance_SubstraitCodec -- Alternative serialization format for expressions