Implementation:Lance format Lance Java Dataset

Knowledge Sources	Lance
Domains	Java_SDK, Dataset_Management
Last Updated	2026-02-08 19:33 GMT

Overview

The Dataset class is the primary entry point for the Lance Java SDK, providing a comprehensive interface for creating, opening, reading, writing, and managing Lance columnar datasets backed by native Rust code via JNI.

Description

Dataset represents a Lance dataset and implements java.io.Closeable for proper resource management of the underlying native handle. It provides static factory methods (write() and open()) that return builder objects for creating or opening datasets. The class supports:

• Dataset creation and writing via the builder pattern (Dataset.write() returning a WriteDatasetBuilder)
• Dataset opening via the builder pattern (Dataset.open() returning an OpenDatasetBuilder)
• Schema management including adding, dropping, and altering columns
• Data scanning through LanceScanner with configurable ScanOptions
• Row-level operations including take (by index), delete (by predicate), and truncate
• Version control with checkout, restore, tagging, and branching
• Indexing including creation, listing, optimization, and statistics retrieval
• Merge insert (upsert) operations via MergeInsertParams
• Compaction and cleanup for storage optimization
• Transactions for atomic multi-step operations
• SQL queries executed through DataFusion
• Blob file access for binary large object columns
• Shallow cloning for creating metadata-only copies of datasets

Thread safety is managed internally through a LockManager that provides read/write lock semantics around all native operations.

Usage

Use this class whenever you need to interact with a Lance dataset from Java. Common scenarios include:

• Creating a new dataset from Arrow data (via ArrowReader or ArrowArrayStream)
• Opening an existing dataset for reading or appending
• Running scans with filters, column projections, nearest-neighbor search, or full-text search
• Managing dataset versions, tags, and branches
• Creating and querying vector or scalar indices
• Performing merge-insert (upsert) operations against existing data

Code Reference

Source Location

Property	Value
File	java/src/main/java/org/lance/Dataset.java
Package	org.lance
Lines	1699

Signature

public class Dataset implements Closeable

Import

import org.lance.Dataset;

I/O Contract

Key Static Methods

Method	Input	Output	Description
write()	none	WriteDatasetBuilder	Returns a builder for writing datasets
open()	none	OpenDatasetBuilder	Returns a builder for opening datasets
drop(String, Map)	path, storageOptions	void	Deletes a dataset at the given path

Key Instance Methods

Method	Input	Output	Description
newScan()	none or ScanOptions	LanceScanner	Creates a dataset scanner
take(List<Long>, List<String>)	indices, columns	ArrowReader	Reads rows by index
delete(String)	predicate	void	Deletes rows matching the predicate
countRows()	none or filter string	long	Counts rows optionally matching a filter
getSchema()	none	Schema	Returns the Arrow schema
getFragments()	none	List<Fragment>	Returns all fragments
createIndex(IndexOptions)	options	Index	Creates an index on the dataset
mergeInsert(MergeInsertParams, ArrowArrayStream)	params, source	MergeInsertResult	Performs upsert operations
version()	none	long	Returns current version ID
checkoutVersion(long)	version	Dataset	Checks out a specific version
compact()	none or CompactionOptions	void	Compacts the dataset
close()	none	void	Releases native resources

Usage Examples

Creating a New Dataset

import org.lance.Dataset;
import org.lance.WriteParams;
import org.apache.arrow.vector.ipc.ArrowReader;

// Using the builder pattern (recommended)
Dataset dataset = Dataset.write()
    .reader(myArrowReader)
    .uri("s3://bucket/table.lance")
    .mode(WriteParams.WriteMode.CREATE)
    .execute();

Opening an Existing Dataset

import org.lance.Dataset;
import org.lance.ReadOptions;

Dataset dataset = Dataset.open()
    .uri("/path/to/dataset.lance")
    .readOptions(new ReadOptions.Builder().build())
    .build();

Scanning Data with Filters

import org.lance.ipc.ScanOptions;
import org.lance.ipc.LanceScanner;
import java.util.Arrays;

ScanOptions options = new ScanOptions.Builder()
    .columns(Arrays.asList("id", "name", "vector"))
    .filter("id > 100")
    .limit(1000)
    .batchSize(512)
    .build();

try (LanceScanner scanner = dataset.newScan(options)) {
    // Process batches from scanner
}

Version Management

// Check current version
long currentVersion = dataset.version();

// Checkout a specific version
Dataset oldVersion = dataset.checkoutVersion(5L);

// Create a tag
dataset.tags().create("release-v1", 5L);

// Checkout by tag
Dataset tagged = dataset.checkoutTag("release-v1");

Merge Insert (Upsert)

import org.lance.merge.MergeInsertParams;
import org.lance.merge.MergeInsertResult;
import org.apache.arrow.c.ArrowArrayStream;

MergeInsertParams params = new MergeInsertParams(Arrays.asList("id"))
    .withMatchedUpdateAll()
    .withNotMatched(MergeInsertParams.WhenNotMatched.InsertAll);

MergeInsertResult result = dataset.mergeInsert(params, sourceStream);
Dataset mergedDataset = result.dataset();

Related Pages

• Lance_format_Lance_Java_Fragment - Fragment-level operations within a dataset
• Lance_format_Lance_Java_WriteDatasetBuilder - Builder for writing datasets
• Lance_format_Lance_Java_ScanOptions - Scan configuration options
• Lance_format_Lance_Java_MergeInsertParams - Merge insert parameters
• Lance_format_Lance_Java_FullTextQuery - Full-text search queries for scans
• Lance_format_Lance_Java_UpdateConfig - Dataset configuration updates
• Heuristic:Lance_format_Lance_Warning_Deprecated_Java_APIs