Implementation:Lance format Lance EncodingTestUtils

Knowledge Sources	Lance
Domains	Encoding, Columnar_Data
Last Updated	2026-02-08 19:33 GMT

Overview

The EncodingTestUtils module provides test infrastructure for round-trip encoding and decoding of Arrow arrays through Lance's encoding pipeline, including simulated I/O, configurable test cases, and data generation helpers.

Description

This module is the primary testing harness for the Lance encoding system. It enables developers to verify that any Arrow data type can be encoded and decoded correctly by simulating the full Lance file write/read cycle in memory.

Key Components:

• SimulatedScheduler -- An implementation of EncodingsIo backed by an in-memory Bytes buffer. It serves byte ranges from the buffer to simulate file I/O without touching the filesystem.

• TestCases -- A configurable test specification that controls:
  • Batch size (default: 100)
  • Page sizes to test (default: [4096, 1MiB])
  • Row ranges to decode (e.g., 0..500, 100..1100, 8000..8500)
  • Index-based takes (e.g., specific row indices [100, 1100, 5000])
  • File version constraints (min/max version to test against)
  • Encoding verification callbacks for asserting specific encoding choices
  • TestCases::basic() provides a standard set of ranges and indices for comprehensive coverage

• ArrayGeneratorProvider (trait) -- Generates test data arrays. Implementations include RandomArrayGeneratorProvider (random data for a given type) and FnArrayGeneratorProvider (custom generator function).

• check_round_trip_encoding_random -- The core test function that:

1. Generates random data using lance-datagen
2. Encodes it through the full encoding pipeline (creating pages and columns)
3. Writes encoded data to an in-memory buffer with proper alignment
4. Creates a SimulatedScheduler for reading
5. Decodes the data back using DecodeBatchScheduler and create_decode_stream
6. Compares decoded output against the original data element by element

• check_basic_random / check_specific_random -- Convenience functions for testing a field with random data and either basic or custom test cases.

Test Infrastructure Details:

• Page buffer alignment is set to MIN_PAGE_BUFFER_ALIGNMENT (8 bytes)
• Maximum page size is 32 MiB
• Tests run across all non-legacy Lance file versions by default (2.0, 2.1, 2.2)
• Null support is version-dependent (e.g., struct nullability only in 2.1+)
• Column indices are computed from the schema, respecting structural vs. legacy encoding differences

Usage

Use this module when:

• Testing a new encoding implementation to ensure round-trip correctness
• Verifying that a data type is properly handled across all Lance file versions
• Writing parameterized tests with specific row ranges, indices, or page sizes

Code Reference

Source Location	rust/lance-encoding/src/testing.rs
Visibility	pub(crate) -- only available within the lance-encoding crate (compiled under #[cfg(test)])
Key Structs	SimulatedScheduler, TestCases
Key Functions	check_round_trip_encoding_random, check_basic_random, check_specific_random, check_round_trip_encoding_generated

I/O Contract

check_round_trip_encoding_random:

Parameter	Type	Description
encoder_factory	impl Fn(LanceFileVersion) -> Box<dyn FieldEncoder>	Creates an encoder for a given version
field	Field	Arrow field describing the data type to test
array_generator_provider	Box<dyn ArrayGeneratorProvider>	Generates test data
test_cases	&TestCases	Specifies which ranges, indices, and versions to test

TestCases Builder Methods:

Method	Description
basic()	Standard test cases with multiple ranges and indices
with_range(range)	Add a row range to decode
with_indices(indices)	Add specific row indices to take
with_batch_size(size)	Set the decode batch size
with_page_sizes(sizes)	Set the page sizes to test
with_min_file_version(v)	Only test versions >= v
with_max_file_version(v)	Only test versions <= v
without_validation()	Skip Arrow data validation

Usage Examples

use lance_encoding::testing::{check_basic_random, check_specific_random, TestCases};
use arrow_schema::{DataType, Field};

// Basic round-trip test for a data type
#[tokio::test]
async fn test_u32_round_trip() {
    let lance_field = lance_core::datatypes::Field::try_from(
        &Field::new("test", DataType::UInt32, true)
    ).unwrap();
    check_basic_random(lance_field).await;
}

// Custom test cases with specific ranges
#[tokio::test]
async fn test_u32_specific_ranges() {
    let lance_field = lance_core::datatypes::Field::try_from(
        &Field::new("test", DataType::UInt32, false)
    ).unwrap();
    let test_cases = TestCases::default()
        .with_range(0..1000)
        .with_range(500..1500)
        .with_indices(vec![0, 500, 999])
        .with_batch_size(256)
        .with_page_sizes(vec![4096]);
    check_specific_random(lance_field, test_cases).await;
}

Related Pages

• Lance_format_Lance_CoreEncoder - The encoding pipeline exercised by round-trip tests
• Lance_format_Lance_BatchDecodeStream - The decoding pipeline exercised by round-trip tests
• Lance_format_Lance_EncodingsIo - SimulatedScheduler implements the EncodingsIo trait
• Lance_format_Lance_DataBlock - Intermediate representation verified during encoding tests