Implementation:Lance format Lance DataGenerator

Knowledge Sources	Lance
Domains	Data_Generation, Testing
Last Updated	2026-02-08 19:33 GMT

Overview

Description

The DataGenerator module in the lance-datagen crate provides a comprehensive framework for generating synthetic Arrow arrays and record batches. It defines the core ArrayGenerator trait along with numerous concrete generator implementations used for testing, benchmarking, and data generation workflows throughout the Lance ecosystem. The module supports generating data of virtually every Arrow data type including primitive types, strings, binaries, lists, structs, maps, fixed-size lists, and dictionary-encoded arrays.

Key abstractions include:

• RowCount, BatchCount, ByteCount, and Dimension -- newtype wrappers providing type safety for generator parameters
• ArrayGenerator trait -- the central trait that all data generators must implement
• CycleNullGenerator and CycleNanGenerator -- decorator generators that overlay null or NaN patterns onto other generators
• Various concrete generators for random, stepped, cycled, and distribution-based data generation

Usage

The DataGenerator module is used primarily in test suites and benchmarks across the Lance crate ecosystem. Generators are composed together to create complex schemas with controlled data distributions. The module integrates with rand and rand_xoshiro for reproducible random number generation via a configurable seed.

Code Reference

Source Location

rust/lance-datagen/src/generator.rs

Signature

pub trait ArrayGenerator: Send + Sync + std::fmt::Debug {
    fn generate(
        &mut self,
        length: RowCount,
        rng: &mut rand_xoshiro::Xoshiro256PlusPlus,
    ) -> Result<Arc<dyn arrow_array::Array>, ArrowError>;

    fn generate_default(
        &mut self,
        length: RowCount,
    ) -> Result<Arc<dyn arrow_array::Array>, ArrowError>;

    fn data_type(&self) -> &DataType;

    fn metadata(&self) -> Option<HashMap<String, String>>;

    fn element_size_bytes(&self) -> Option<ByteCount>;
}

Import

use lance_datagen::generator::{ArrayGenerator, RowCount, BatchCount, ByteCount, Dimension};

I/O Contract

Inputs

Parameter	Type	Description
length	RowCount	Number of rows (elements) to generate in the output array
rng	&mut Xoshiro256PlusPlus	Seeded random number generator for reproducible output

Outputs

Type	Description
Result<Arc<dyn Array>, ArrowError>	A generated Arrow array of the requested length, or an error if generation fails
&DataType	The Arrow data type produced by this generator (via data_type())
Option<ByteCount>	The fixed element size in bytes, or None for variable-width types

Usage Examples

use lance_datagen::{gen_batch, array};
use arrow_schema::DataType;
use arrow_array::types::Int32Type;

// Generate a batch with stepped integer column and random float column
let batch = gen_batch()
    .col("id", array::step::<Int32Type>())
    .col("value", array::rand::<Float32Type>())
    .into_batch(RowCount::from(100))
    .unwrap();

// Generate with null pattern overlay
let generator = array::step::<Int32Type>()
    .with_nulls(vec![true, true, false]); // every 3rd element is null

Related Pages

• Lance_format_Lance_CrateRoot -- Main Lance crate that uses datagen in tests
• Lance_format_Lance_AccumulationQueue -- Encoding utility that may process generated data in tests