Implementation:Lance format Lance IndexExtension

Knowledge Sources	Lance
Domains	Indexing, Infrastructure
Last Updated	2026-02-08 19:33 GMT

Overview

Description

The IndexExtension module defines the extension point traits for plugging custom index implementations into the Lance dataset system. It provides a three-level trait hierarchy:

• IndexExtension -- Base trait requiring Send + Sync + DeepSizeOf. Provides:
  • index_type() -- Returns the IndexType (Vector or Scalar)
  • to_generic() -- Upcast workaround for Arc<Self> to Arc<dyn IndexExtension>
  • to_scalar() -- Downcast attempt to ScalarIndexExtension
  • to_vector() -- Downcast attempt to VectorIndexExtension

• ScalarIndexExtension -- Extends IndexExtension for scalar (B-tree, inverted) indices. Currently a marker trait with planned future methods.

• VectorIndexExtension -- Extends IndexExtension for vector similarity indices. Provides two async methods:
  • create_index -- Creates a vector index on a dataset column given index parameters
  • load_index -- Loads an existing vector index from a file reader

The module includes a comprehensive test suite with a MockIndex implementation that validates the extension mechanism works correctly with the dataset pipeline.

Usage

Index extensions are registered with the Lance Session and invoked during Dataset::create_index and index loading operations. Custom index implementations (e.g., IVF-PQ, HNSW) implement these traits to integrate with Lance's index management system.

Code Reference

Source Location

rust/lance/src/session/index_extension.rs

Signature

pub trait IndexExtension: Send + Sync + DeepSizeOf {
    fn index_type(&self) -> IndexType;
    fn to_generic(self: Arc<Self>) -> Arc<dyn IndexExtension>;
    fn to_scalar(self: Arc<Self>) -> Option<Arc<dyn ScalarIndexExtension>>;
    fn to_vector(self: Arc<Self>) -> Option<Arc<dyn VectorIndexExtension>>;
}

pub trait ScalarIndexExtension: IndexExtension {
    // Planned: scalar index creation and loading methods
}

#[async_trait::async_trait]
pub trait VectorIndexExtension: IndexExtension {
    async fn create_index(
        &self,
        dataset: &Dataset,
        column: &str,
        uuid: &str,
        params: &dyn IndexParams,
    ) -> Result<()>;

    async fn load_index(
        &self,
        dataset: Arc<Dataset>,
        column: &str,
        uuid: &str,
        reader: PreviousFileReader,
    ) -> Result<Arc<dyn VectorIndex>>;
}

Import

use lance::session::index_extension::{
    IndexExtension, ScalarIndexExtension, VectorIndexExtension,
};

I/O Contract

Inputs

Parameter	Type	Description
dataset	&Dataset / Arc<Dataset>	The dataset to create or load indices for
column	&str	Name of the column to index
uuid	&str	Unique identifier for the index
params	&dyn IndexParams	Index-specific configuration parameters (e.g., IVF-PQ settings)
reader	PreviousFileReader	File reader for loading an existing index from storage

Outputs

Type	Description
IndexType	The type of index (Vector or Scalar)
Result<()>	Success indicator for index creation
Result<Arc<dyn VectorIndex>>	A loaded vector index ready for search operations
Option<Arc<dyn ScalarIndexExtension>>	Downcast result for scalar index extensions
Option<Arc<dyn VectorIndexExtension>>	Downcast result for vector index extensions

Usage Examples

use lance::session::index_extension::VectorIndexExtension;
use lance::index::vector::VectorIndexParams;

// Implement the trait for a custom index type
struct MyVectorIndex;

#[async_trait::async_trait]
impl VectorIndexExtension for MyVectorIndex {
    async fn create_index(
        &self,
        dataset: &Dataset,
        column: &str,
        uuid: &str,
        params: &dyn IndexParams,
    ) -> Result<()> {
        // Custom index creation logic
        Ok(())
    }

    async fn load_index(
        &self,
        dataset: Arc<Dataset>,
        column: &str,
        uuid: &str,
        reader: PreviousFileReader,
    ) -> Result<Arc<dyn VectorIndex>> {
        // Custom index loading logic
        todo!()
    }
}

Related Pages

• Lance_format_Lance_IndexCaches -- Cache layer for index data
• Lance_format_Lance_MemWalIndex -- MemWAL index management functions
• Lance_format_Lance_CrateRoot -- Main crate containing the session/index modules