Implementation:Lance format Lance Java StorageOptionsProvider

Knowledge Sources	Lance
Domains	Java_SDK, Dataset_Management
Last Updated	2026-02-08 19:33 GMT

Overview

Description

The StorageOptionsProvider interface defines a contract for supplying cloud storage credentials to Lance datasets. It enables automatic credential refresh for long-running operations on cloud storage backends (S3, Azure Blob Storage, GCS). Currently this is primarily used for refreshing AWS temporary access credentials obtained through STS or similar services.

The interface declares two methods: fetchStorageOptions() (required), which returns a flat map of credential key-value pairs with an optional expires_at_millis timestamp, and providerId() (default), which returns a human-readable identifier used for equality comparison and ObjectStore caching.

Usage

Implement StorageOptionsProvider to integrate with custom credential management systems. Pass the provider to ReadOptions.Builder.setStorageOptionsProvider() when opening a dataset. The provider is called automatically before credentials expire, ensuring uninterrupted access during long-running queries, training jobs, or data processing pipelines.

Code Reference

Source Location

java/src/main/java/org/lance/io/StorageOptionsProvider.java

Signature

public interface StorageOptionsProvider {
    Map<String, String> fetchStorageOptions();
    default String providerId();
}

Import

import org.lance.io.StorageOptionsProvider;

I/O Contract

fetchStorageOptions() Return Map Keys

Key	Required	Description
expires_at_millis	Recommended	Unix timestamp in milliseconds (as string) when credentials expire; triggers automatic refresh
aws_access_key_id	For S3	AWS access key ID
aws_secret_access_key	For S3	AWS secret access key
aws_session_token	For S3	AWS session token (optional for permanent credentials)
account_name	For Azure	Azure storage account name
account_key / sas_token	For Azure	Azure account key or SAS token
service_account_key / token	For GCS	GCP service account key or OAuth token

Outputs

Method	Return Type	Description
fetchStorageOptions()	Map<String, String>	Flat map of credential key-value pairs
providerId()	String	Human-readable identifier for caching and equality; defaults to class name + toString()

Usage Examples

// Custom implementation for AWS STS credentials
public class StsStorageProvider implements StorageOptionsProvider {
    @Override
    public Map<String, String> fetchStorageOptions() {
        Map<String, String> creds = new HashMap<>();
        creds.put("aws_access_key_id", "ASIA...");
        creds.put("aws_secret_access_key", "secret");
        creds.put("aws_session_token", "token");
        creds.put("expires_at_millis",
            String.valueOf(System.currentTimeMillis() + 3600000L));
        return creds;
    }
}

// Use with a dataset
StorageOptionsProvider provider = new StsStorageProvider();
Dataset dataset = Dataset.open(
    "s3://bucket/table.lance",
    new ReadOptions.Builder()
        .setStorageOptionsProvider(provider)
        .build()
);

Related Pages

• Lance_format_Lance_Java_NamespaceStorageOptions -- Namespace-backed implementation that fetches credentials via describeTable()
• Lance_format_Lance_Java_RestAdapter -- REST adapter for namespace testing