Principle:Lance format Lance Vector Data Preparation
| Knowledge Sources | |
|---|---|
| Domains | Vector_Search, Data_Engineering |
| Last Updated | 2026-02-08 19:00 GMT |
Overview
Vector Data Preparation is the process of enriching a Lance dataset with vector (embedding) columns so that the data can be indexed and searched using approximate nearest neighbor algorithms.
Description
Before any vector search can be performed on a Lance dataset, the data must contain one or more vector columns. These columns store fixed-dimensional arrays of floating-point numbers (embeddings) that represent the semantic or structural content of each row. The preparation step bridges the gap between raw tabular data and vector-search-ready data.
Lance provides the schema evolution mechanism to add new columns to an existing dataset without rewriting the original data. The add_columns API accepts several forms of column transforms:
- BatchUDF -- a user-defined function that receives batches of existing data and returns a new
RecordBatchcontaining the vector column. This is the most flexible approach and is commonly used to call an embedding model in-process. - SqlExpressions -- SQL expressions that derive new columns from existing ones. Useful for lightweight numeric transforms.
- Stream / Reader -- precomputed vectors supplied as a
SendableRecordBatchStreamorRecordBatchReader. Ideal when embeddings have already been computed externally. - AllNulls -- creates the column with null values, to be filled later.
Vector columns must be typed as DataType::FixedSizeList(Field::new("item", DataType::Float32, true), dim) where dim is the embedding dimensionality. Other element types such as Float16, Float64, and UInt8 are also supported.
Usage
Use vector data preparation whenever:
- You have a Lance dataset of text, images, or other modalities and need to add embedding columns for similarity search.
- You want to add a new embedding model's output alongside an existing vector column.
- You receive precomputed embeddings from an external service and need to merge them into the dataset.
- You are evolving the schema of a dataset that was originally created without vector columns.
Theoretical Basis
Embedding Representation
An embedding is a mapping f : X -> R^d from a high-dimensional input space X (text, images, audio) into a d-dimensional real vector space. The key property is that semantically similar inputs are mapped to nearby points under a chosen distance metric (L2, cosine, or dot product).
In columnar storage, each row's embedding is stored as a fixed-size list of d scalar values. This layout enables:
- Contiguous memory access -- all dimensions of a single vector are stored together, which is cache-friendly for distance computations.
- Batch processing -- entire columns of vectors can be loaded and processed in SIMD-friendly batches.
- Zero-copy interop -- the Arrow
FixedSizeListtype maps directly to contiguous memory, avoiding serialization overhead.
Schema Evolution Model
Lance's schema evolution operates as a metadata-first operation. When add_columns is called:
- The transform is applied to every fragment in the dataset, producing new data files that contain only the new columns.
- A new manifest is created that merges the original schema with the new column schema.
- The transaction is committed atomically, so readers either see the old schema or the fully updated schema.
This approach avoids rewriting existing data and ensures that adding a vector column is an O(n) operation in the number of rows, not O(n * m) where m is the number of existing columns.