Implementation:Lance format Lance MemWalIndex

Knowledge Sources	Lance
Domains	Indexing, Infrastructure
Last Updated	2026-02-08 19:33 GMT

Overview

Description

The MemWalIndex module provides functions for managing the in-memory Write-Ahead Log (MemWAL) index in Lance datasets. The MemWAL index is stored as part of the dataset's index metadata and tracks:

• Configuration -- Region specifications and maintained indexes
• Merge progress -- Which generations have been merged to the base table per region
• Region state snapshots -- Eventually consistent state of each region

Writers do not update the index on every write; instead they update region manifests directly. This module provides three core functions:

• load_mem_wal_index_details -- Deserializes MemWalIndexDetails from an IndexMetadata entry, validating that the protobuf type URL ends with "MemWalIndexDetails"
• open_mem_wal_index -- Loads details and wraps them in an Arc<MemWalIndex>
• update_mem_wal_index_merged_generations -- Atomically updates merged generation records during merge-insert commits. For each region, it keeps the higher generation number. If no MemWAL index exists yet, it creates one.
• new_mem_wal_index_meta -- Creates a new IndexMetadata entry with a fresh UUID, the MemWAL index name, serialized protobuf details, and current timestamp

Usage

This module is used internally during merge-insert commit operations to track which WAL generations have been incorporated into the base table. It enables concurrent writers to coordinate through the index metadata.

Code Reference

Source Location

rust/lance/src/index/mem_wal.rs

Signature

pub(crate) fn load_mem_wal_index_details(index: IndexMetadata) -> Result<MemWalIndexDetails>;

pub(crate) fn open_mem_wal_index(index: IndexMetadata) -> Result<Arc<MemWalIndex>>;

pub(crate) fn update_mem_wal_index_merged_generations(
    indices: &mut Vec<IndexMetadata>,
    dataset_version: u64,
    new_merged_generations: Vec<MergedGeneration>,
) -> Result<()>;

pub(crate) fn new_mem_wal_index_meta(
    dataset_version: u64,
    details: MemWalIndexDetails,
) -> Result<IndexMetadata>;

Import

// Internal (crate-private) module
use crate::index::mem_wal::{
    load_mem_wal_index_details, open_mem_wal_index,
    update_mem_wal_index_merged_generations, new_mem_wal_index_meta,
};

I/O Contract

Inputs

Parameter	Type	Description
index	IndexMetadata	An index metadata entry containing serialized MemWalIndexDetails as a protobuf Any
indices	&mut Vec<IndexMetadata>	Mutable list of all dataset indices, modified in-place
dataset_version	u64	The dataset version number associated with the new index entry
new_merged_generations	Vec<MergedGeneration>	List of region/generation pairs to record as merged

Outputs

Type	Description
Result<MemWalIndexDetails>	Deserialized MemWAL index configuration and merge state
Result<Arc<MemWalIndex>>	A shared reference to the opened MemWAL index
Result<()>	Success indicator for generation update (modifies indices in-place)
Result<IndexMetadata>	A new index metadata entry with fresh UUID and serialized details

Usage Examples

use lance_index::mem_wal::MergedGeneration;

// During a merge-insert commit, update which generations are merged
let mut indices = dataset.load_indices().await?;
update_mem_wal_index_merged_generations(
    &mut indices,
    dataset.version().version,
    vec![
        MergedGeneration { region_id: 0, generation: 5 },
        MergedGeneration { region_id: 1, generation: 3 },
    ],
)?;

Related Pages

• Lance_format_Lance_IndexCaches -- Cache layer for index metadata
• Lance_format_Lance_IndexExtension -- Extension traits for creating and loading indices