Implementation:Lance format Lance JsonUdf

Knowledge Sources	Lance
Domains	DataFusion_Integration, Query_Execution
Last Updated	2026-02-08 19:33 GMT

Overview

The JsonUdf module implements a suite of DataFusion user-defined functions (UDFs) for querying and extracting data from JSONB (binary JSON) columns stored as LargeBinary Arrow arrays.

Description

This module provides a comprehensive set of JSON operations that work on Lance's JSONB column format (backed by the jsonb crate). JSONB data is stored as LargeBinary arrays, and all functions accept a JSONB column as the first argument and typically a JSONPath or field key as the second.

JSON Type System:

• JsonbType -- An enum representing the type of a JSONB value: Null (0), Boolean (1), Int64 (2), Float64 (3), String (4), Array (5), Object (6). Used for type-tagged extraction.

Helper Module (common):

• KeyType -- An enum (Field/Index) for pre-parsed JSON field access, optimizing repeated field/index lookups.
• columnar_to_arrays -- Converts ColumnarValue arguments to ArrayRef vectors, handling scalar broadcast.
• Validation helpers: validate_arg_count, extract_jsonb_array, extract_string_array.
• get_json_value_by_key -- Retrieves a JSON field or array element using a pre-parsed key type.

Registered UDFs:

UDF Name	Signature	Return Type	Description
json_extract	(LargeBinary, Utf8) -> Utf8	Utf8	Extracts a value at a JSONPath and returns it as a JSON string
json_extract_with_type	(LargeBinary, Utf8) -> Struct	Struct{value: LargeBinary, type_tag: UInt8}	Extracts a value with its JSONB type tag for type-aware processing
json_exists	(LargeBinary, Utf8) -> Boolean	Boolean	Checks whether a path exists in the JSONB document
json_get	(LargeBinary, Utf8) -> LargeBinary	LargeBinary	Gets a JSONB sub-document at the given key/index
json_get_string	(LargeBinary, Utf8) -> Utf8	Utf8	Extracts a string value at the given key/index
json_get_int	(LargeBinary, Utf8) -> Int64	Int64	Extracts an integer value at the given key/index
json_get_float	(LargeBinary, Utf8) -> Float64	Float64	Extracts a float value at the given key/index
json_get_bool	(LargeBinary, Utf8) -> Boolean	Boolean	Extracts a boolean value at the given key/index
json_array_contains	(LargeBinary, Utf8, Utf8) -> Boolean	Boolean	Checks if a JSON array at a path contains a specified value
json_array_length	(LargeBinary, Utf8) -> Int64	Int64	Returns the length of a JSON array at the given path

Usage

Use these UDFs when querying Lance datasets that contain JSONB columns. They are automatically registered when a Lance session context is created via register_functions.

Code Reference

Source Location

rust/lance-datafusion/src/udf/json.rs

Signature

pub fn json_extract_udf() -> ScalarUDF
pub fn json_extract_with_type_udf() -> ScalarUDF
pub fn json_exists_udf() -> ScalarUDF
pub fn json_get_udf() -> ScalarUDF
pub fn json_get_string_udf() -> ScalarUDF
pub fn json_get_int_udf() -> ScalarUDF
pub fn json_get_float_udf() -> ScalarUDF
pub fn json_get_bool_udf() -> ScalarUDF
pub fn json_array_contains_udf() -> ScalarUDF
pub fn json_array_length_udf() -> ScalarUDF

Import

use lance_datafusion::udf::json::{
    json_extract_udf, json_exists_udf, json_get_udf,
    json_get_string_udf, json_get_int_udf, json_get_float_udf,
    json_get_bool_udf, json_array_contains_udf, json_array_length_udf,
};

I/O Contract

Input	Type	Description
jsonb_column	LargeBinary	The JSONB data stored as a LargeBinary Arrow array
path / key	Utf8	A JSONPath string or field name / array index
value (json_array_contains)	Utf8	The value to search for within a JSON array

Output	Type	Description
Varies by function	See UDF table above	Extracted values in the appropriate Arrow type, with nulls for missing paths or type mismatches

Usage Examples

// In SQL queries against a Lance dataset with a JSONB column "data":
// Extract a nested string field
SELECT json_get_string(data, 'name') FROM my_table;

// Check if a path exists
SELECT * FROM my_table WHERE json_exists(data, '$.address.city');

// Get array length
SELECT json_array_length(data, 'items') FROM my_table;

// Check array containment
SELECT * FROM my_table WHERE json_array_contains(data, 'tags', 'important');

Related Pages

• Lance_format_Lance_UdfRegistration -- Registration module that registers all JSON UDFs with the session context
• Lance_format_Lance_FilterPlanner -- Planner that can use JSON UDFs in filter expressions