Implementation:Lance format Lance BlobEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

BlobStructuralEncoder is a logical encoding that stores large binary data (blobs) in external out-of-line buffers and records inline descriptors containing each blob's position and size.

Description

The blob encoding handles large binary (LargeBinary) arrays by writing the actual binary payload to external buffers outside the normal page structure. For each blob value, it creates a descriptor struct with two fields: position: u64 and size: u64. These descriptors are encoded inline using a PrimitiveStructuralEncoder with packed struct metadata. The encoder also manages repetition/definition levels (rep/def) for nullable blob columns, wrapping the resulting encode tasks with blob-specific layout metadata via ProtobufUtils21::blob_layout.

Usage

Use this encoding when storing large binary objects (images, documents, serialized models) in Lance files. The blob encoding is selected automatically for columns with the LargeBinary data type and the blob kind annotation. It is particularly suited for values that are too large to store efficiently inline within the page structure.

Code Reference

Source Location	Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/logical/blob.rs, Lines: 1-800
Signature	pub struct BlobStructuralEncoder { descriptor_encoder: Box<dyn FieldEncoder>, def_meaning: Option<Arc<[DefinitionInterpretation]>>, } impl BlobStructuralEncoder { pub fn new( field: &Field, column_index: u32, options: &crate::encoder::EncodingOptions, compression_strategy: Arc<dyn crate::compression::CompressionStrategy>, ) -> Result<Self>; }
Import	use lance_encoding::encodings::logical::blob::BlobStructuralEncoder;

I/O Contract

Direction	Type	Description
Input	ArrayRef (LargeBinary)	Large binary array containing blob data
Input	OutOfLineBuffers	External buffer store for out-of-line blob data
Input	RepDefBuilder	Repetition/definition level builder for nullability
Output	Vec<EncodeTask>	Encode tasks producing pages with blob descriptors (position, size)
Output	EncodedColumn	Column with external blob buffers and inline descriptor pages

Usage Examples

use lance_encoding::encodings::logical::blob::BlobStructuralEncoder;
use lance_core::datatypes::Field;
use std::sync::Arc;

// BlobStructuralEncoder is typically instantiated by the encoding framework
// when it detects a LargeBinary column with blob annotation.
let encoder = BlobStructuralEncoder::new(
    &field,
    column_index,
    &encoding_options,
    compression_strategy.clone(),
)?;

Related Pages

• Implementation:Lance_format_Lance_PrimitiveBlobDecoding - Decoding routines for blob data
• Implementation:Lance_format_Lance_PrimitiveEncoding - The primitive structural encoder used for descriptors
• Implementation:Lance_format_Lance_PackedEncoding - Packed struct encoding used for blob descriptors