Implementation:Lance format Lance SafeScalarValue

Knowledge Sources	Lance
Domains	DataFusion_Integration, Query_Execution
Last Updated	2026-02-08 19:33 GMT

Overview

The SafeScalarValue module provides safe type coercion for DataFusion ScalarValue literals, handling the lack of automatic type coercion when converting SQL strings to logical expressions.

Description

When DataFusion parses SQL strings into logical expressions, no automatic type coercion is applied to literal values. For example, x = 7 always produces x = 7_u64 regardless of column x's actual type. This module bridges that gap with:

• safe_coerce_scalar -- The primary function that safely coerces a ScalarValue to a target DataType. It handles all numeric type conversions (Int8 through UInt64, Float32, Float64), string-to-string conversions, date/time conversions (Date32 to timestamps, Utf8 to Date32/Date64/Timestamp), and decimal types. Conversions that would overflow or lose precision return None rather than panicking.
• cast_scalar_value -- An internal helper that uses Arrow's compute cast kernel to perform scalar value casting via a single-element array round-trip.

The coercion logic is comprehensive, covering:

• All integer narrowing/widening with overflow checking via TryFrom
• Integer-to-float promotion (with a note about inherent precision loss for large i32/i64 to f32)
• String literal parsing for date and timestamp types
• Date-to-timestamp conversion using millisecond arithmetic
• Decimal type casting via DataFusion's built-in cast_to

Usage

Use this module when constructing filter expressions from SQL strings that need to match a Lance schema's column types, particularly in the filter planner and logical expression resolution pipeline.

Code Reference

Source Location

rust/lance-datafusion/src/expr.rs

Signature

pub fn safe_coerce_scalar(value: &ScalarValue, ty: &DataType) -> Option<ScalarValue>

Import

use lance_datafusion::expr::safe_coerce_scalar;

I/O Contract

Input	Type	Description
value	&ScalarValue	The DataFusion scalar value to be coerced
ty	&DataType	The target Arrow DataType to coerce to

Output	Type	Description
result	Option<ScalarValue>	The coerced scalar value, or None if the conversion is not possible or would overflow

Usage Examples

use datafusion::scalar::ScalarValue;
use arrow_schema::DataType;
use lance_datafusion::expr::safe_coerce_scalar;

// Coerce an Int64 literal to Int32
let value = ScalarValue::Int64(Some(42));
let coerced = safe_coerce_scalar(&value, &DataType::Int32);
assert_eq!(coerced, Some(ScalarValue::Int32(Some(42))));

// Overflow returns None
let big = ScalarValue::Int64(Some(i64::MAX));
let coerced = safe_coerce_scalar(&big, &DataType::Int8);
assert_eq!(coerced, None);

// String to Date32
let date_str = ScalarValue::Utf8(Some("2024-01-15".to_string()));
let coerced = safe_coerce_scalar(&date_str, &DataType::Date32);
assert!(coerced.is_some());

Related Pages

• Lance_format_Lance_LogicalExpr -- Uses safe_coerce_scalar for expression resolution
• Lance_format_Lance_FilterPlanner -- Planner that relies on scalar coercion for filter construction