Implementation:Lance format Lance MemWalWriteBench

Knowledge Sources	Lance
Domains	Benchmarking, Performance
Last Updated	2026-02-08 19:33 GMT

Overview

Description

MemWalWriteBench is a Criterion-based benchmark that measures write throughput for the Lance MemWAL (Memory Write-Ahead Log) subsystem. It writes batches of records through the MemWAL pipeline and measures the time taken to complete all writes, exercising different combinations of:

• Durable writes — Whether writes wait for WAL flush before returning.
• Indexed writes — Whether writes update indexes synchronously.
• Index maintenance — Which indexes are maintained in-flight (BTree, FTS, IVF-PQ).

The benchmark creates a base dataset with configurable schema (id, text, and vector columns), creates indexes on it (BTree on id, FTS on text, IVF-PQ on vector), and then measures the throughput of writing batches through the MemWAL layer with various configurations.

The data schema includes:

• id (Int64) — Primary key with BTree index
• text (Utf8) — Text column with full-text search index
• vector (FixedSizeList<Float32>) — Vector column with IVF-PQ index

Usage

This benchmark is used to evaluate write throughput under various MemWAL configurations, helping developers understand the write-side overhead of durability guarantees and index maintenance. It is particularly useful for tuning WAL buffer sizes, flush intervals, and MemTable size thresholds.

Code Reference

Source Location

rust/lance/benches/mem_wal_write.rs (673 lines)

Signature

The benchmark defines a single benchmark target:

fn bench_lance_memwal_write(c: &mut Criterion)

Import

use lance::dataset::mem_wal::{DatasetMemWalExt, MemWalConfig, RegionWriterConfig};
use lance::dataset::{Dataset, WriteParams};
use lance::index::vector::VectorIndexParams;
use lance_index::scalar::ScalarIndexParams;
use lance_index::vector::ivf::IvfBuildParams;
use lance_index::vector::pq::PQBuildParams;
use lance_index::{DatasetIndexExt, IndexType};
use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion, Throughput};

I/O Contract

Inputs

Parameter	Type	Default	Description
DATASET_PREFIX	Environment variable	Temp directory	Base URI for datasets (S3, GCS, Azure, or local path)
BATCH_SIZE	Environment variable	20	Number of rows per write batch
NUM_BATCHES	Environment variable	1000	Total number of batches to write
DURABLE_WRITE	Environment variable	no	Whether writes wait for WAL flush (yes/no/both)
INDEXED_WRITE	Environment variable	no	Whether writes update indexes synchronously (yes/no/both)
MAX_WAL_BUFFER_SIZE	Environment variable	1 MB	WAL buffer size in bytes
MAX_FLUSH_INTERVAL_MS	Environment variable	1000	WAL flush interval in milliseconds (0 to disable)
MAX_MEMTABLE_SIZE	Environment variable	64 MB	MemTable size threshold in bytes
VECTOR_DIM	Environment variable	512	Vector dimension for the vector column
MEMWAL_MAINTAINED_INDEXES	Environment variable	id_btree	Comma-separated list of index names to maintain (id_btree, text_fts, vector_ivfpq, or none)
SAMPLE_SIZE	Environment variable	10	Number of benchmark iterations (minimum 10)

Outputs

Output	Type	Description
Criterion report	HTML/JSON	Write throughput statistics (bytes/sec), with optional flamegraph profiling on Linux
Console output	Text	Prints configuration summary, batch sizes, and throughput metrics during execution

Usage Examples

Run the benchmark with default settings against a local temp directory:

cargo bench -p lance --bench mem_wal_write

Run against S3 storage:

export AWS_DEFAULT_REGION=us-east-1
export DATASET_PREFIX=s3://your-bucket/bench/mem_wal
cargo bench -p lance --bench mem_wal_write

Run with custom configuration to test durable indexed writes:

export BATCH_SIZE=50
export NUM_BATCHES=500
export DURABLE_WRITE=yes
export INDEXED_WRITE=yes
export MEMWAL_MAINTAINED_INDEXES=id_btree,text_fts
cargo bench -p lance --bench mem_wal_write

Compare durable vs non-durable writes:

export DURABLE_WRITE=both
cargo bench -p lance --bench mem_wal_write

Related Pages

• Lance_format_Lance_MemWalReadBench — Benchmark for LSM scanner read performance with MemWAL
• Lance_format_Lance_MemtableReadBench — Benchmark comparing MemTable vs in-memory Lance table read performance