Implementation:Lance format Lance BlobEncoding

Knowledge Sources	Lance
Domains	Encoding, Compression
Last Updated	2026-02-08 19:33 GMT

Overview

BlobStructuralEncoder is a logical encoding that stores large binary data (blobs) in external out-of-line buffers and records inline descriptors containing each blob's position and size.

Description

The blob encoding handles large binary (LargeBinary) arrays by writing the actual binary payload to external buffers outside the normal page structure. For each blob value, it creates a descriptor struct with two fields: position: u64 and size: u64. These descriptors are encoded inline using a PrimitiveStructuralEncoder with packed struct metadata. The encoder also manages repetition/definition levels (rep/def) for nullable blob columns, wrapping the resulting encode tasks with blob-specific layout metadata via ProtobufUtils21::blob_layout.

Usage

Use this encoding when storing large binary objects (images, documents, serialized models) in Lance files. The blob encoding is selected automatically for columns with the LargeBinary data type and the blob kind annotation. It is particularly suited for values that are too large to store efficiently inline within the page structure.

Code Reference

Source Location Repository: lance-format/lance, File: rust/lance-encoding/src/encodings/logical/blob.rs, Lines: 1-800

Signature

pub struct BlobStructuralEncoder {
    descriptor_encoder: Box<dyn FieldEncoder>,
    def_meaning: Option<Arc<[DefinitionInterpretation]>>,
}

impl BlobStructuralEncoder {
    pub fn new(
        field: &Field,
        column_index: u32,
        options: &crate::encoder::EncodingOptions,
        compression_strategy: Arc<dyn crate::compression::CompressionStrategy>,
    ) -> Result<Self>;
}

Import use lance_encoding::encodings::logical::blob::BlobStructuralEncoder;

I/O Contract

Direction	Type	Description
Input	`ArrayRef` (LargeBinary)	Large binary array containing blob data
Input	`OutOfLineBuffers`	External buffer store for out-of-line blob data
Input	`RepDefBuilder`	Repetition/definition level builder for nullability
Output	`Vec<EncodeTask>`	Encode tasks producing pages with blob descriptors (position, size)
Output	`EncodedColumn`	Column with external blob buffers and inline descriptor pages

Usage Examples

use lance_encoding::encodings::logical::blob::BlobStructuralEncoder;
use lance_core::datatypes::Field;
use std::sync::Arc;

// BlobStructuralEncoder is typically instantiated by the encoding framework
// when it detects a LargeBinary column with blob annotation.
let encoder = BlobStructuralEncoder::new(
    &field,
    column_index,
    &encoding_options,
    compression_strategy.clone(),
)?;

Related Pages

Lance_format_Lance_PrimitiveBlobDecoding - Decoding routines for blob data
Lance_format_Lance_PrimitiveEncoding - The primitive structural encoder used for descriptors
Lance_format_Lance_PackedEncoding - Packed struct encoding used for blob descriptors

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment