Implementation:Lance format Lance DataBlock

Knowledge Sources	Lance
Domains	Encoding, Columnar_Data
Last Updated	2026-02-08 19:33 GMT

Overview

DataBlock is an enum representing physical data layouts (fixed-width, variable-width, struct, dictionary, etc.) that sits between Arrow's ArrayData and Array, serving as the intermediate representation exchanged between encoders and decoders.

Description

The DataBlock type system fills a gap in the Arrow ecosystem. Arrow's ArrayData is a generic collection of buffers that does not describe what those buffers represent, while Arrow's Array is too specific because it cares about logical data types. DataBlock captures the physical layout of encoded data independent of its logical type.

Variants:

• Empty -- No data (zero-length array)
• Constant -- A single value repeated num_values times
• AllNull -- All values are null (no buffers needed)
• Nullable -- Wraps another data block with a validity bitmap (deprecated in 2.1)
• FixedWidth -- A single data buffer with a fixed number of bits per value (booleans, integers, floats, etc.)
• FixedSizeList -- A child data block with a fixed dimension per element
• VariableWidth -- An offsets buffer plus a data buffer (strings, binary data)
• Opaque -- Arbitrary buffers with no regular structure (cannot be directly converted to Arrow)
• Struct -- A collection of child data blocks with optional validity
• Dictionary -- An indices block plus a dictionary block

Additional restrictions over Arrow: Offset buffers in VariableWidthBlock MUST start at 0, which normalizes the representation and simplifies encoder/decoder development.

Each variant stores a BlockInfo for computed statistics (bit width, data size, cardinality, run count, etc.) that drive compression decisions.

Usage

Use DataBlock when:

• Building a new encoder or decoder that transforms data at the physical level
• Converting between Arrow arrays and the Lance physical representation
• Chaining decoders together where the output of one decoder feeds into another

Code Reference

Source Location	rust/lance-encoding/src/data.rs
Primary Enum	DataBlock
Key Structs	FixedWidthDataBlock, VariableWidthBlock, StructDataBlock, DictionaryDataBlock, ConstantDataBlock, BlockInfo
Import	use lance_encoding::data::{DataBlock, FixedWidthDataBlock, VariableWidthBlock};

I/O Contract

DataBlock Conversion Methods:

Method	Input	Output	Notes
into_arrow(data_type, validate)	self, DataType, bool	Result<ArrayData>	Zero-copy for most variants
into_buffers()	self	Vec<LanceBuffer>	For serialization
try_clone()	&self	Result<DataBlock>	Reference-counted clone
data_size()	&self	u64	Total byte size
num_values()	&self	u64	Number of logical values

FixedWidthDataBlock Fields:

Field	Type	Description
data	LanceBuffer	The raw data buffer
bits_per_value	u64	Number of bits per element
num_values	u64	Number of elements
block_info	BlockInfo	Computed statistics

VariableWidthBlock Fields:

Field	Type	Description
data	LanceBuffer	The concatenated value bytes
offsets	LanceBuffer	The offsets buffer (starts at 0)
bits_per_offset	u8	32 or 64
num_values	u64	Number of elements

Usage Examples

use lance_encoding::data::{DataBlock, FixedWidthDataBlock, BlockInfo};
use lance_encoding::buffer::LanceBuffer;
use arrow_schema::DataType;

// Create a fixed-width data block of u32 values
let values = vec![1u32, 2, 3, 4];
let buf = LanceBuffer::reinterpret_vec(values);
let block = DataBlock::FixedWidth(FixedWidthDataBlock {
    data: buf,
    bits_per_value: 32,
    num_values: 4,
    block_info: BlockInfo::new(),
});

// Convert to Arrow
let arrow_data = block.into_arrow(DataType::UInt32, false).unwrap();
assert_eq!(arrow_data.len(), 4);

Related Pages

• Lance_format_Lance_LanceBuffer - The buffer type stored within every data block variant
• Lance_format_Lance_Statistics - The ComputeStat and GetStat traits that operate on data blocks
• Lance_format_Lance_Compression_Traits - Compression strategies inspect data block statistics to choose algorithms
• Lance_format_Lance_BatchDecodeStream - Decoders produce DataBlock instances during the decode phase
• Lance_format_Lance_CoreEncoder - Encoders consume Arrow arrays and produce data blocks