Implementation:Lance format Lance SpillMechanism

Knowledge Sources	Lance
Domains	DataFusion_Integration, Query_Execution
Last Updated	2026-02-08 19:33 GMT

Overview

The SpillMechanism module implements a replay-able spill-to-disk strategy for Arrow record batch streams, allowing data to be buffered in memory up to a configurable limit and then spilled to a temporary file that can be read multiple times.

Description

This module provides a producer-consumer pattern for managing large intermediate datasets that may exceed available memory. Key components include:

• create_replay_spill -- The main entry point that creates a (SpillSender, SpillReceiver) pair. Data is initially buffered in memory; if the memory limit is exceeded, all buffered data is flushed to a file via Arrow IPC streaming format, and subsequent writes go directly to the file.

• SpillSender -- The writer side of the spill. Key methods:
  • write(batch) -- Writes a RecordBatch to the spill. Data accumulates in memory until the memory limit is reached, at which point it transitions to file-based storage using Arrow IPC StreamWriter.
  • finish() -- Signals that writing is complete. Must be called to allow readers to know the stream has ended.
  • On drop, the temporary file is deleted.

• SpillReceiver -- The reader side of the spill. It is Clone, allowing multiple consumers:
  • read() -- Returns a SendableRecordBatchStream that emits all batches from the spill. The stream can be opened before, during, or after writing. It waits for new data using a tokio::sync::watch channel.

• SpillReader -- Internal reader state machine that handles two modes:
  • Buffered mode -- Reads from the in-memory batch array when data never exceeded the memory limit.
  • File mode -- Opens an Arrow IPC StreamReader on the spill file, skipping batches already read during the buffered phase.

• AsyncStreamReader -- An internal wrapper that reads Arrow IPC batches from a file using blocking I/O on a Tokio spawn_blocking thread.

The communication between sender and receiver uses a tokio::sync::watch channel carrying a WriteStatus that tracks the number of batches written, the data location (buffered or on disk), error state, and completion flag.

Usage

Use this module when you need to:

• Buffer intermediate query results that may be read multiple times (e.g., for join build sides)
• Limit memory consumption during operations that produce large intermediate datasets
• Provide a replay-able stream of data without keeping everything in memory

Code Reference

Source Location

rust/lance-datafusion/src/spill.rs

Signature

pub fn create_replay_spill(
    path: std::path::PathBuf,
    schema: Arc<Schema>,
    memory_limit: usize,
) -> (SpillSender, SpillReceiver)

pub struct SpillSender { /* ... */ }
impl SpillSender {
    pub fn write(&mut self, batch: &RecordBatch) -> Result<(), DataFusionError>;
    pub fn finish(self) -> Result<(), DataFusionError>;
}

#[derive(Clone)]
pub struct SpillReceiver { /* ... */ }
impl SpillReceiver {
    pub fn read(&self) -> SendableRecordBatchStream;
}

Import

use lance_datafusion::spill::{create_replay_spill, SpillSender, SpillReceiver};

I/O Contract

Input	Type	Description
path	PathBuf	Path for the temporary spill file (should not already exist)
schema	Arc<Schema>	Arrow schema for the record batches
memory_limit	usize	Maximum bytes to buffer in memory before spilling to disk

Output	Type	Description
SpillSender	struct	Writer handle for sending batches to the spill
SpillReceiver	struct (Clone)	Reader handle that can produce multiple replay-able streams
SpillReceiver::read()	SendableRecordBatchStream	A stream of all batches written to the spill

Usage Examples

use lance_datafusion::spill::create_replay_spill;
use std::sync::Arc;

let schema = Arc::new(arrow_schema);
let (mut sender, receiver) = create_replay_spill(
    "/tmp/spill_file.arrow".into(),
    schema,
    64 * 1024 * 1024, // 64 MB memory limit
);

// Write batches
sender.write(&batch1)?;
sender.write(&batch2)?;
sender.finish()?;

// Read all batches (can be called multiple times)
let stream1 = receiver.read();
let stream2 = receiver.read(); // Replay from the beginning

Related Pages

• Implementation:Lance_format_Lance_ExecPlans -- Execution options that enable spilling via LanceExecutionOptions
• Implementation:Lance_format_Lance_Chunker -- Batch chunking utilities that may precede spill operations