Implementation:Lance format Lance LanceBuffer

Knowledge Sources	Lance
Domains	Encoding, Columnar_Data
Last Updated	2026-02-08 19:33 GMT

Overview

LanceBuffer is a copy-on-write byte buffer wrapper around Arrow's Buffer that provides cheap cloning, zero-copy slicing, and automatic memory alignment for passing encoded data throughout the Lance encoding pipeline.

Description

LanceBuffer is the fundamental byte buffer type used across the Lance encoding system. It wraps arrow_buffer::Buffer to provide reference-counted, zero-copy operations. Key capabilities include:

• Zero-copy construction from Vec<u8>, Buffer, bytes::Bytes (when aligned), and Vec<T> for any Arrow-native type
• Type reinterpretation via reinterpret_vec, borrow_to_typed_slice, and borrow_to_typed_view which safely cast raw bytes into typed scalar buffers
• Buffer combination through concat, concat_into_one, and zip_into_one for merging multiple buffers
• Bit-level slicing via bit_slice_le_with_length using Arrow's bitwise little-endian convention
• Alignment-aware construction from external byte sources, performing a copy only when necessary to satisfy alignment requirements

The buffer ensures that data passed between encoders and decoders is always in a consistent, efficiently accessible format. It implements Deref<Target=[u8]>, AsRef<[u8]>, and IntoIterator for ergonomic access.

Usage

Use LanceBuffer whenever you need to:

• Pass byte data between encoding and decoding stages
• Convert between typed vectors and raw byte representations for serialization
• Slice buffers for page-level or block-level data access
• Combine multiple encoded buffers into a single contiguous buffer for writing

Code Reference

Source Location	rust/lance-encoding/src/buffer.rs
Primary Struct	LanceBuffer
Import	use lance_encoding::buffer::LanceBuffer;

I/O Contract

Key Construction Methods:

Method	Input	Output	Zero-Copy
from(Vec<u8>)	Vec<u8>	LanceBuffer	Yes
from(Buffer)	arrow_buffer::Buffer	LanceBuffer	Yes
from_bytes(bytes, bytes_per_value)	bytes::Bytes, alignment	LanceBuffer	Yes (if aligned)
reinterpret_vec(vec)	Vec<T: ArrowNativeType>	LanceBuffer	Yes
copy_slice(slice)	&[u8]	LanceBuffer	No (copies)

Key Transformation Methods:

Method	Input	Output	Notes
into_buffer()	self	arrow_buffer::Buffer	Never copies
borrow_to_typed_slice::<T>()	&self	ScalarBuffer<T>	Copies if not aligned
borrow_to_typed_view::<T>()	&self	Cow<[T]>	Copies if not aligned
concat_into_one(buffers)	Vec<LanceBuffer>	LanceBuffer	Single buffer returned as-is
zip_into_one(buffers, num_values)	Vec<(LanceBuffer, u64)>	Result<LanceBuffer>	Interleaves values

Usage Examples

use lance_encoding::buffer::LanceBuffer;

// Create from a Vec<u8>
let buf = LanceBuffer::from(vec![1u8, 2, 3, 4]);
assert_eq!(buf.len(), 4);

// Create from typed data (zero-copy)
let typed_data = vec![1u32, 2, 3];
let buf = LanceBuffer::reinterpret_vec(typed_data);
assert_eq!(buf.len(), 12); // 3 * 4 bytes

// Reinterpret back to typed slice
let scalar: arrow_buffer::ScalarBuffer<u32> = buf.borrow_to_typed_slice::<u32>();
assert_eq!(scalar.as_ref(), &[1u32, 2, 3]);

// Concatenate multiple buffers
let buf1 = LanceBuffer::from(vec![1u8, 2]);
let buf2 = LanceBuffer::from(vec![3u8, 4]);
let combined = LanceBuffer::concat_into_one(vec![buf1, buf2]);
assert_eq!(combined.as_ref(), &[1, 2, 3, 4]);

// Create utility buffers
let zeros = LanceBuffer::all_unset(4);   // 4 bytes of 0x00
let ones = LanceBuffer::all_set(4);      // 4 bytes of 0xFF
let empty = LanceBuffer::empty();        // 0 bytes

// Slice a buffer (zero-copy)
let buf = LanceBuffer::from(vec![10u8, 20, 30, 40, 50]);
let sliced = buf.slice_with_length(1, 3);
assert_eq!(sliced.as_ref(), &[20, 30, 40]);

Related Pages

• Lance_format_Lance_DataBlock - Uses LanceBuffer as the underlying storage for all data block variants
• Lance_format_Lance_Compression_Traits - Compression routines consume and produce LanceBuffer instances
• Lance_format_Lance_CoreEncoder - Encoded pages contain Vec<LanceBuffer> for their data
• Lance_format_Lance_RepDef - Rep-def serialization produces LanceBuffer output