Principle:Lance format Lance Vector Index Configuration
| Knowledge Sources | |
|---|---|
| Domains | Vector_Search, Indexing |
| Last Updated | 2026-02-08 19:00 GMT |
Overview
Vector Index Configuration is the process of selecting and parameterizing the algorithmic stages (partitioning, graph construction, quantization) that define how a vector index is built and queried.
Description
Building an efficient vector index requires choosing among multiple algorithmic strategies, each with their own trade-offs between recall, latency, memory usage, and build time. Lance's vector index system is designed as a pipeline of stages, where each stage applies a specific algorithm:
- IVF (Inverted File Index) -- partitions the vector space into clusters using k-means. At query time, only a subset of partitions (controlled by
nprobes) are searched, dramatically reducing the search space. - HNSW (Hierarchical Navigable Small World) -- builds a multi-layer proximity graph enabling logarithmic-time nearest neighbor traversal. Controlled by parameters
m(graph connectivity),ef_construction(build-time accuracy), andmax_level(hierarchy depth). - PQ (Product Quantization) -- compresses vectors by splitting them into sub-vectors and quantizing each independently. Reduces memory footprint at the cost of some distance approximation error.
- SQ (Scalar Quantization) -- quantizes each dimension independently to fewer bits (typically 8). Simpler than PQ with a different accuracy/compression trade-off.
- RQ (Residual Quantization) -- applies quantization to residuals iteratively for finer approximation.
The distance metric (L2, Cosine, Dot) determines how similarity is measured and must be consistent between indexing and querying.
Usage
Configure a vector index when:
- You are about to build a vector index on a Lance dataset column and need to decide on the index type.
- You need to tune recall vs. latency trade-offs for a specific workload.
- You are scaling to a larger dataset and need to adjust partition counts or quantization parameters.
Theoretical Basis
IVF Partitioning
IVF uses k-means clustering to partition n vectors into k clusters. Each vector is assigned to the nearest centroid. At query time, the query vector is compared to all centroids, and only the nprobes nearest partitions are searched.
The expected search cost is approximately O(k + nprobes * n/k), compared to O(n) for brute force. The optimal number of partitions typically scales as sqrt(n).
Key parameters:
- num_partitions -- number of IVF clusters (typically sqrt(n) to 4*sqrt(n))
- max_iters -- k-means iterations (default: 50)
- sample_rate -- ratio of training samples to centroids (default: 256)
HNSW Graph
HNSW constructs a multi-layer skip-list-like graph. The bottom layer contains all vectors; each successive layer contains a random subset. Search begins at the top layer and descends, using greedy traversal at each level.
Key parameters:
- m -- number of bi-directional edges per node (default: 20). Higher values increase recall but use more memory.
- ef_construction -- size of dynamic candidate list during build (default: 150). Higher values produce a better graph at the cost of build time.
- max_level -- maximum number of layers (default: 7).
Product Quantization
PQ decomposes each d-dimensional vector into M sub-vectors of dimension d/M, then independently quantizes each sub-vector into one of 2^b centroids (where b is the number of bits, typically 8).
This reduces storage from d * 4 bytes (float32) to M bytes per vector, a compression ratio of 4d/M.
Key parameters:
- num_sub_vectors -- number of PQ sub-vector splits (default: 16)
- num_bits -- bits per sub-quantizer (default: 8, yielding 256 centroids)
Composite Index Types
Lance supports combining these stages into composite indices:
| Index Type | Stages | Characteristics |
|---|---|---|
| IVF_FLAT | IVF | Fast build, exact distances within partitions, highest memory |
| IVF_PQ | IVF + PQ | Good compression, approximate distances, moderate recall |
| IVF_HNSW_PQ | IVF + HNSW + PQ | Best recall/latency trade-off for large datasets |
| IVF_HNSW_SQ | IVF + HNSW + SQ | Similar to IVF_HNSW_PQ with simpler quantization |
| IVF_SQ | IVF + SQ | Simpler alternative to IVF_PQ |