Implementation:Lance format Lance ExecPlans

Knowledge Sources	Lance
Domains	DataFusion_Integration, Query_Execution
Last Updated	2026-02-08 19:33 GMT

Overview

The ExecPlans module provides DataFusion execution plan nodes, session context management, and plan execution utilities for Lance's query engine.

Description

This module is the central hub for running DataFusion physical plans within Lance. Key components include:

• OneShotExec -- A source execution plan node that wraps an existing SendableRecordBatchStream. It can only be executed once; subsequent executions return an error. It implements the full ExecutionPlan trait including display formatting.
• LanceExecutionOptions -- A configuration struct controlling execution behavior: memory pool size, spill-to-disk settings, batch size, target partition count, and an optional statistics callback. Supports environment variable overrides (LANCE_MEM_POOL_SIZE, LANCE_MAX_TEMP_DIRECTORY_SIZE, LANCE_BYPASS_SPILLING).
• get_session_context -- Creates or retrieves a cached SessionContext configured with Lance's UDFs, memory pools, and disk managers. Uses an LRU cache keyed by the resolved execution options.
• new_session_context -- Creates a fresh SessionContext without caching.
• execute_plan -- Executes a physical plan on a single partition, returning a SendableRecordBatchStream. Logs plan details, attaches tracing spans, and reports summary metrics (IOPS, bytes read, etc.) when the stream completes.
• analyze_plan -- Wraps a plan in DataFusion's AnalyzeExec to collect execution metrics and returns them as a formatted string.
• ExecutionSummaryCounts -- A struct summarizing execution metrics: IOPS, requests, bytes read, indices loaded, partitions loaded, and index comparisons.
• StrictBatchSizeExec -- An execution plan wrapper that enforces exact output batch sizes using StrictBatchSizeStream.
• SessionContextExt -- A trait extending SessionContext to execute SQL on a stream of record batches via execute_sql_on_stream.

Usage

Use this module when you need to:

• Execute DataFusion physical plans within the Lance runtime
• Create one-shot stream-backed execution nodes for scan operations
• Configure and manage DataFusion session contexts with Lance-specific settings
• Collect and report execution metrics for performance monitoring

Code Reference

Source Location

rust/lance-datafusion/src/exec.rs

Signature

pub struct OneShotExec { /* ... */ }
impl OneShotExec {
    pub fn new(stream: SendableRecordBatchStream) -> Self;
    pub fn from_batch(batch: RecordBatch) -> Self;
}

pub struct LanceExecutionOptions {
    pub use_spilling: bool,
    pub mem_pool_size: Option<u64>,
    pub max_temp_directory_size: Option<u64>,
    pub batch_size: Option<usize>,
    pub target_partition: Option<usize>,
    pub execution_stats_callback: Option<ExecutionStatsCallback>,
    pub skip_logging: bool,
}

pub fn get_session_context(options: &LanceExecutionOptions) -> SessionContext;
pub fn new_session_context(options: &LanceExecutionOptions) -> SessionContext;

pub fn execute_plan(
    plan: Arc<dyn ExecutionPlan>,
    options: LanceExecutionOptions,
) -> Result<SendableRecordBatchStream>;

pub async fn analyze_plan(
    plan: Arc<dyn ExecutionPlan>,
    options: LanceExecutionOptions,
) -> Result<String>;

Import

use lance_datafusion::exec::{
    OneShotExec, LanceExecutionOptions, execute_plan, get_session_context,
    ExecutionSummaryCounts, StrictBatchSizeExec,
};

I/O Contract

Input	Type	Description
plan	Arc<dyn ExecutionPlan>	A DataFusion physical execution plan to run
options	LanceExecutionOptions	Execution configuration (memory limits, batch size, spilling, etc.)

Output	Type	Description
execute_plan	Result<SendableRecordBatchStream>	A stream of RecordBatches produced by executing the plan on partition 0
analyze_plan	Result<String>	A formatted string containing execution plan metrics and statistics

Usage Examples

use lance_datafusion::exec::{OneShotExec, execute_plan, LanceExecutionOptions};
use std::sync::Arc;

// Create a one-shot execution node from an existing stream
let exec = OneShotExec::new(my_stream);

// Or from a single RecordBatch
let exec = OneShotExec::from_batch(batch);

// Execute with default options
let result_stream = execute_plan(
    Arc::new(exec),
    LanceExecutionOptions::default(),
)?;

// Execute with custom options
let options = LanceExecutionOptions {
    use_spilling: true,
    batch_size: Some(8192),
    ..Default::default()
};
let result_stream = execute_plan(Arc::new(plan), options)?;

Related Pages

• Lance_format_Lance_Chunker -- Batch chunking utilities used with execution plans
• Lance_format_Lance_FilterPlanner -- SQL filter planning that produces plans for execution
• Lance_format_Lance_UdfRegistration -- UDF registration called during session context creation
• Lance_format_Lance_SpillMechanism -- Spill-to-disk support enabled via LanceExecutionOptions