Implementation:Lance format Lance Dataset Add Columns For Vectors

Knowledge Sources	Lance
Domains	Vector_Search, Data_Engineering
Last Updated	2026-02-08 19:00 GMT

Overview

Concrete tool for appending new vector (embedding) columns to an existing Lance dataset, provided by the Lance library's schema evolution API.

Description

Dataset::add_columns is the primary entry point for enriching a Lance dataset with new columns, including vector columns. It accepts a NewColumnTransform enum that describes how the new column values are produced. The operation processes every fragment in the dataset, writes new data files containing only the new columns, and atomically commits a merged schema. This is analogous to ALTER TABLE ADD COLUMN in SQL but optimized for the Lance columnar format.

The underlying implementation delegates to schema_evolution::add_columns, which iterates over all fragments, applies the transform, and creates a Merge transaction that is committed in a single atomic step.

Usage

Use this API when you need to:

• Add embedding columns computed by a UDF (e.g., calling an embedding model on text fields).
• Merge precomputed vectors from an external pipeline into the dataset.
• Derive new numeric columns from existing ones via SQL expressions.

Code Reference

Source Location

• Repository: Lance
• File: rust/lance/src/dataset.rs
• Lines: L2578-L2585 (public API), internals at rust/lance/src/dataset/schema_evolution.rs L408-L430

Signature

impl Dataset {
    pub async fn add_columns(
        &mut self,
        transforms: NewColumnTransform,
        read_columns: Option<Vec<String>>,
        batch_size: Option<u32>,
    ) -> Result<()>
}

Import

use lance::dataset::Dataset;
use lance::dataset::schema_evolution::{NewColumnTransform, BatchUDF};

I/O Contract

Inputs

Name	Type	Required	Description
transforms	NewColumnTransform	Yes	Defines how new column values are produced. Variants: BatchUDF (closure receiving existing data, returning new columns), SqlExpressions (Vec of (name, SQL) pairs), Stream (SendableRecordBatchStream of precomputed data), Reader (RecordBatchReader of precomputed data), AllNulls (schema for null-initialized columns).
read_columns	Option<Vec<String>>	No	Columns from the existing dataset to pass to the transform function. If None, all columns are read. For BatchUDF transforms, specifying only the needed columns improves performance.
batch_size	Option<u32>	No	Number of rows per batch passed to the transform. If None, the default fragment batch size is used.

Outputs

Name	Type	Description
result	Result<()>	Returns Ok(()) on success. The dataset is mutated in place with the new schema and data files. A new version is committed atomically.

Usage Examples

Adding a vector column with a BatchUDF

use std::sync::Arc;
use arrow_schema::{Schema as ArrowSchema, Field, DataType};
use arrow_array::{RecordBatch, FixedSizeListArray, Float32Array};
use lance::dataset::Dataset;
use lance::dataset::schema_evolution::{NewColumnTransform, BatchUDF};

async fn add_embeddings(dataset: &mut Dataset) -> lance::Result<()> {
    let dim = 128i32;
    let output_schema = Arc::new(ArrowSchema::new(vec![
        Field::new(
            "vector",
            DataType::FixedSizeList(
                Arc::new(Field::new("item", DataType::Float32, true)),
                dim,
            ),
            false,
        ),
    ]));

    let udf = BatchUDF {
        mapper: Box::new(move |batch: &RecordBatch| {
            let num_rows = batch.num_rows();
            // Replace with real embedding computation
            let values = Float32Array::from(vec![0.0f32; num_rows * dim as usize]);
            let list = FixedSizeListArray::try_new_from_values(values, dim)?;
            Ok(RecordBatch::try_new(
                output_schema.clone(),
                vec![Arc::new(list)],
            )?)
        }),
        output_schema: output_schema.clone(),
        result_checkpoint: None,
    };

    dataset
        .add_columns(
            NewColumnTransform::BatchUDF(udf),
            Some(vec!["text".to_string()]),  // only read the text column
            None,
        )
        .await
}

Adding a column with SQL expressions

use lance::dataset::Dataset;
use lance::dataset::schema_evolution::NewColumnTransform;

async fn add_derived_column(dataset: &mut Dataset) -> lance::Result<()> {
    dataset
        .add_columns(
            NewColumnTransform::SqlExpressions(vec![
                ("norm_score".to_string(), "score / 100.0".to_string()),
            ]),
            None,
            None,
        )
        .await
}

Related Pages

Implements Principle

• Principle:Lance_format_Lance_Vector_Data_Preparation