Workflow:Lance format Lance Vector Search Pipeline
| Knowledge Sources | |
|---|---|
| Domains | Vector_Search, ML_Ops, Information_Retrieval |
| Last Updated | 2026-02-08 19:00 GMT |
Overview
End-to-end process for building vector indices on embedding columns and performing approximate nearest neighbor (ANN) search with optional metadata filtering in a Lance dataset.
Description
This workflow covers the complete vector search pipeline from data preparation through index construction to query execution. Lance supports multiple vector index types including IVF-PQ (Inverted File with Product Quantization), IVF-HNSW-SQ (Hierarchical Navigable Small World with Scalar Quantization), and flat indices. The pipeline handles index training on sampled data, partitioning vectors into clusters, encoding vectors with quantization, and executing search queries with configurable recall-latency tradeoffs. Results can be refined by post-filtering with SQL predicates or increasing the refine factor for higher recall.
Usage
Execute this workflow when you have a dataset containing vector embeddings (e.g., text embeddings, image features, audio fingerprints) and need sub-millisecond approximate nearest neighbor search. This is the standard path for building recommendation systems, semantic search engines, image similarity retrieval, and RAG (Retrieval-Augmented Generation) pipelines.
Execution Steps
Step 1: Vector Data Preparation
Prepare the dataset with a vector column containing fixed-dimension embeddings. Vectors are stored as FixedSizeList columns in Arrow format. If vectors are not yet present, add an embedding column using schema evolution with a Python UDF that calls an embedding model. Ensure all vectors have the same dimensionality and compatible data types (Float32 or Float16).
Key considerations:
- All vectors in a column must have identical dimensionality
- Float16 vectors reduce storage by 50% with minimal recall impact
- Normalize vectors before indexing if using cosine distance
- Verify vector column exists and has correct type before indexing
Step 2: Index Configuration
Select the appropriate index type and configure its parameters based on the dataset size, dimensionality, and recall-latency requirements. The main index types are IVF-PQ for large datasets (millions of vectors), IVF-HNSW-SQ for balanced recall and performance, and flat indices for small datasets or exact search.
Configuration parameters:
- num_partitions - Number of IVF clusters (typically sqrt(num_rows))
- num_sub_vectors - PQ sub-vector count (higher = better recall, more memory)
- distance_type - L2 (Euclidean), Cosine, or Dot product
- max_iterations - K-means training iterations
- sample_rate - Fraction of data used for training centroids
Step 3: Index Training and Building
Train the vector index on the dataset. This involves sampling data for centroid training, running K-means clustering to establish IVF partitions, training quantization codebooks (PQ/SQ), and writing the index structure to storage. For HNSW indices, this also builds the navigable small-world graph within each partition.
Key considerations:
- Training samples a subset of data; larger samples improve centroid quality
- Index building reads all vectors and assigns them to partitions
- The process is CPU and memory intensive; use appropriate hardware
- Index metadata is recorded in the dataset manifest as a new version
Step 4: Query Execution
Execute nearest neighbor queries by providing a query vector and the number of results (k). The search process probes a subset of IVF partitions (controlled by nprobes), computes approximate distances using quantized representations, and returns the top-k candidates. Increasing nprobes improves recall at the cost of latency.
Key considerations:
- Default nprobes is 1; increase to 10-50 for better recall
- Use refine_factor to re-rank candidates with full-precision vectors
- Combine vector search with SQL filters for hybrid queries
- Results include the distance score and requested metadata columns
Step 5: Result Filtering and Refinement
Apply post-query refinement to improve result quality. This includes re-ranking with full-precision distance computation (refine_factor), applying SQL WHERE predicates to filter results by metadata, projecting specific output columns, and limiting result count. For hybrid search combining vector similarity with keyword matching, compose vector and full-text search queries.
Key considerations:
- Refine factor of 5-10 significantly improves recall with moderate overhead
- Pre-filtering narrows the search space before ANN; post-filtering applies after
- Project only needed columns to minimize data transfer
- Monitor recall using ground truth benchmarks to tune parameters