Implementation:Lance format Lance ProjectionPlan

Knowledge Sources	Lance
Domains	DataFusion_Integration, Query_Execution
Last Updated	2026-02-08 19:33 GMT

Overview

The ProjectionPlan module translates user-requested column selections and computed expressions into a physical projection that determines which columns to load from disk and how to transform them.

Description

This module bridges the gap between user-specified column selections (potentially including SQL expressions) and the physical I/O required to satisfy them. Key components include:

• ProjectionPlan -- The main struct containing:
  • physical_projection -- A Projection specifying which columns to load from the dataset, including system column flags (_rowid, _rowaddr, _row_offset, version tracking columns).
  • must_add_row_offset -- Whether row addresses need conversion to row offsets.
  • requested_output_expr -- A list of OutputColumn entries, each pairing an output name with a DataFusion Expr.

• ProjectionPlan::from_expressions -- Builds a plan from SQL expression pairs (output_name, expression_string). Supports wildcards, system column references, and computed columns.
• ProjectionPlan::from_schema -- Builds a plan from a Lance Schema, supporting partial nested projections (e.g., selecting only specific fields within a struct column). System columns in the schema are handled via flags rather than physical projection.
• ProjectionPlan::full -- Creates a plan that projects all columns from the dataset.
• ProjectionPlan::to_physical_exprs -- Converts the output expressions to physical expressions suitable for DataFusion's ProjectionExec.
• ProjectionPlan::execute -- Applies the projection to a stream of record batches by creating and executing a DataFusion ProjectionExec node.

• ProjectionBuilder -- An internal builder that accumulates column specifications, resolves system columns, tracks physical column dependencies, and constructs the final ProjectionPlan.
• OutputColumn -- A simple struct pairing an Expr with its output column name.

Usage

Use ProjectionPlan when you need to:

• Determine which physical columns must be loaded from a Lance dataset to satisfy a query
• Support computed columns (SQL expressions) in scan projections
• Handle system columns (_rowid, _rowaddr) alongside data columns
• Apply final column transformations to scanned data

Code Reference

Source Location

rust/lance-datafusion/src/projection.rs

Signature

#[derive(Clone, Debug)]
pub struct OutputColumn {
    pub expr: Expr,
    pub name: String,
}

#[derive(Clone, Debug)]
pub struct ProjectionPlan {
    pub physical_projection: Projection,
    pub must_add_row_offset: bool,
    pub requested_output_expr: Vec<OutputColumn>,
}

impl ProjectionPlan {
    pub fn from_expressions(
        base: Arc<dyn Projectable>,
        columns: &[(impl AsRef<str>, impl AsRef<str>)],
    ) -> Result<Self>;

    pub fn from_schema(
        base: Arc<dyn Projectable>,
        projection: &Schema,
    ) -> Result<Self>;

    pub fn full(base: Arc<dyn Projectable>) -> Result<Self>;

    pub fn to_physical_exprs(
        &self,
        current_schema: &ArrowSchema,
    ) -> Result<Vec<(Arc<dyn PhysicalExpr>, String)>>;
}

Import

use lance_datafusion::projection::{ProjectionPlan, OutputColumn};

I/O Contract

Input	Type	Description
base	Arc<dyn Projectable>	The dataset schema source providing available columns
columns	&[(impl AsRef<str>, impl AsRef<str>)]	Pairs of (output_name, sql_expression) for expression-based projection
projection	&Schema	A Lance schema for schema-based projection (may include partial nested fields)

Output	Type	Description
ProjectionPlan	struct	Contains the physical columns to load, system column flags, and output expressions
to_physical_exprs	Result<Vec<(Arc<dyn PhysicalExpr>, String)>>	Physical expressions and their output names for DataFusion ProjectionExec

Usage Examples

use lance_datafusion::projection::ProjectionPlan;
use std::sync::Arc;

// Project specific columns with computed expressions
let plan = ProjectionPlan::from_expressions(
    dataset_schema.clone(),
    &[
        ("id", "id"),
        ("full_name", "first_name || ' ' || last_name"),
        ("_rowid", "_rowid"),
    ],
)?;

// Project from a schema subset
let plan = ProjectionPlan::from_schema(dataset_schema.clone(), &subset_schema)?;

// Full projection of all columns
let plan = ProjectionPlan::full(dataset_schema)?;

Related Pages

• Lance_format_Lance_FilterPlanner -- Planner used internally for parsing SQL expressions in projections
• Lance_format_Lance_ExecPlans -- Execution infrastructure for running the projection plan
• Lance_format_Lance_LogicalExpr -- Logical expression utilities used during column resolution