Implementation:Lance format Lance LanceDataFrame

Knowledge Sources	Lance
Domains	DataFusion, Infrastructure
Last Updated	2026-02-08 19:33 GMT

Overview

Description

The LanceDataFrame module (dataframe.rs) provides the LanceTableProvider struct and the SessionContextExt trait for integrating Lance datasets with Apache DataFusion's query engine.

LanceTableProvider is a full-featured TableProvider implementation that supports:

• Projection pushdown -- Only reads requested columns from the Lance dataset
• Filter pushdown -- Reports all filters as exactly applicable (TableProviderFilterPushDown::Exact) since DataFusion handles the actual filtering
• Limit pushdown -- Passes row limits through to the Lance scanner
• System columns -- Optionally includes _rowid and _rowaddr pseudo-columns in the table schema
• Ordered/unordered scanning -- Configurable via new_with_ordering

SessionContextExt extends DataFusion's SessionContext with three convenience methods:

• read_lance -- Creates an ordered DataFrame over a Lance dataset
• read_lance_unordered -- Creates an unordered DataFrame for parallel-friendly scanning
• read_one_shot -- Creates a DataFrame from a SendableRecordBatchStream that can only be consumed once

OneShotPartitionStream is an internal helper that wraps a stream in an Arc<Mutex>, implementing DataFusion's PartitionStream trait for single-use streaming.

Usage

This module is the primary way to use SQL queries over Lance datasets. It is used internally throughout the Lance codebase and is also available for end users who want to integrate Lance with DataFusion-based query pipelines.

Code Reference

Source Location

rust/lance/src/datafusion/dataframe.rs

Signature

pub struct LanceTableProvider { /* ... */ }

impl LanceTableProvider {
    pub fn new(dataset: Arc<Dataset>, with_row_id: bool, with_row_addr: bool) -> Self;
    pub fn new_with_ordering(
        dataset: Arc<Dataset>, with_row_id: bool, with_row_addr: bool, ordered: bool,
    ) -> Self;
    pub fn dataset(&self) -> Arc<Dataset>;
}

pub trait SessionContextExt {
    fn read_lance(&self, dataset: Arc<Dataset>, with_row_id: bool, with_row_addr: bool)
        -> datafusion::common::Result<DataFrame>;
    fn read_lance_unordered(&self, dataset: Arc<Dataset>, with_row_id: bool, with_row_addr: bool)
        -> datafusion::common::Result<DataFrame>;
    fn read_one_shot(&self, data: SendableRecordBatchStream)
        -> datafusion::common::Result<DataFrame>;
}

pub struct OneShotPartitionStream { /* ... */ }

Import

use lance::datafusion::{LanceTableProvider, SessionContextExt, OneShotPartitionStream};

I/O Contract

Inputs

Parameter	Type	Description
dataset	Arc<Dataset>	Lance dataset to expose as a DataFusion table
with_row_id	bool	Include _rowid pseudo-column in schema
with_row_addr	bool	Include _rowaddr pseudo-column in schema
ordered	bool	Whether results should be returned in deterministic order
data	SendableRecordBatchStream	A one-shot record batch stream for read_one_shot

Outputs

Type	Description
Arc<dyn ExecutionPlan>	DataFusion execution plan from scan()
DataFrame	DataFusion DataFrame for SQL query composition
Vec<TableProviderFilterPushDown>	All filters reported as Exact

Usage Examples

use lance::datafusion::{LanceTableProvider, SessionContextExt};
use datafusion::prelude::SessionContext;
use std::sync::Arc;

let ctx = SessionContext::new();

// Register with system columns
ctx.register_table(
    "my_table",
    Arc::new(LanceTableProvider::new(Arc::new(dataset), true, true)),
)?;

// SQL query with filter and limit pushdown
let df = ctx.sql("SELECT _rowid, name FROM my_table WHERE age > 25 LIMIT 100").await?;
let results = df.collect().await?;

// Convenience method
let df = ctx.read_lance(Arc::new(dataset), false, false)?;

Related Pages

• Lance_format_Lance_LanceTableProvider -- The logical_plan.rs implementation (simpler variant)
• Lance_format_Lance_CrateRoot -- Main crate that exports the datafusion module
• Lance_format_Lance_SessionCaches -- Caching layer used by the dataset during scans