Implementation:Lance format Lance Java CompactionOptions

Knowledge Sources	Lance
Domains	Java_SDK, Dataset_Management
Last Updated	2026-02-08 19:33 GMT

Overview

Description

CompactionOptions is a serializable configuration class for controlling Lance dataset compaction behavior. It defines parameters for target fragment size, row group limits, file size limits, deletion materialization, thread count, batch size, and index remap deferral. All fields are Optional to avoid conflicting with default values defined in the Rust compaction code. The class implements custom Java serialization via writeObject/readObject to handle Optional fields, enabling transmission across network boundaries in distributed computing frameworks.

Usage

CompactionOptions is constructed via its static builder() method and inner Builder class. Instances are passed to Compaction.planCompaction() and Compaction.commitCompaction(), as well as stored within CompactionTask for distributed execution. The serialization support makes it compatible with frameworks like Apache Spark.

Code Reference

Source Location

java/src/main/java/org/lance/compaction/CompactionOptions.java

Signature

public class CompactionOptions implements Serializable {
    public static Builder builder();
    public Optional<Long> getTargetRowsPerFragment();
    public Optional<Long> getMaxRowsPerGroup();
    public Optional<Long> getMaxBytesPerFile();
    public Optional<Boolean> getMaterializeDeletions();
    public Optional<Float> getMaterializeDeletionsThreshold();
    public Optional<Long> getNumThreads();
    public Optional<Long> getBatchSize();
    public Optional<Boolean> getDeferIndexRemap();
    public String toString();

    public static class Builder {
        public Builder withTargetRowsPerFragment(long targetRowsPerFragment);
        public Builder withMaxRowsPerGroup(long maxRowsPerGroup);
        public Builder withMaxBytesPerFile(long maxBytesPerFile);
        public Builder withMaterializeDeletions(boolean materializeDeletions);
        public Builder withMaterializeDeletionsThreshold(float threshold);
        public Builder withNumThreads(long numThreads);
        public Builder withBatchSize(long batchSize);
        public Builder withDeferIndexRemap(boolean deferIndexRemap);
        public CompactionOptions build();
    }
}

Import

import org.lance.compaction.CompactionOptions;

I/O Contract

**Builder Parameters**
Method	Type	Description
withTargetRowsPerFragment()	`long`	Target number of rows per compacted fragment
withMaxRowsPerGroup()	`long`	Maximum rows per row group within a file
withMaxBytesPerFile()	`long`	Maximum file size in bytes for compacted files
withMaterializeDeletions()	`boolean`	Whether to rewrite fragments to remove deleted rows
withMaterializeDeletionsThreshold()	`float`	Fraction of deleted rows that triggers materialization (0.0 to 1.0)
withNumThreads()	`long`	Number of threads for compaction execution
withBatchSize()	`long`	Batch size for reading during compaction
withDeferIndexRemap()	`boolean`	Whether to defer index remapping to a separate step

**Getter Return Values**
Method	Return Type	Description
getTargetRowsPerFragment()	`Optional<Long>`	Target rows per fragment or empty
getMaxRowsPerGroup()	`Optional<Long>`	Max rows per group or empty
getMaxBytesPerFile()	`Optional<Long>`	Max bytes per file or empty
getMaterializeDeletions()	`Optional<Boolean>`	Materialization flag or empty
getMaterializeDeletionsThreshold()	`Optional<Float>`	Materialization threshold or empty
getNumThreads()	`Optional<Long>`	Thread count or empty
getBatchSize()	`Optional<Long>`	Batch size or empty
getDeferIndexRemap()	`Optional<Boolean>`	Defer index remap flag or empty

Usage Examples

import org.lance.compaction.CompactionOptions;

// Basic compaction with target fragment size
CompactionOptions options = CompactionOptions.builder()
    .withTargetRowsPerFragment(1024 * 1024)
    .withMaxRowsPerGroup(1024)
    .build();

// Full configuration for production workload
CompactionOptions options = CompactionOptions.builder()
    .withTargetRowsPerFragment(500000)
    .withMaxRowsPerGroup(2048)
    .withMaxBytesPerFile(512L * 1024 * 1024)
    .withMaterializeDeletions(true)
    .withMaterializeDeletionsThreshold(0.1f)
    .withNumThreads(8)
    .withBatchSize(4096)
    .withDeferIndexRemap(false)
    .build();

// Minimal options (all Rust defaults)
CompactionOptions defaults = CompactionOptions.builder().build();

// Serialization for distributed computing (e.g., Spark)
// CompactionOptions implements Serializable
ObjectOutputStream out = new ObjectOutputStream(stream);
out.writeObject(options);

Related Pages

Lance_format_Lance_Java_Compaction - Static methods that consume CompactionOptions for planning and committing
Lance_format_Lance_Java_CompactionTask - Tasks that carry CompactionOptions for distributed execution
Lance_format_Lance_Java_CleanupPolicy - Complementary cleanup configuration for post-compaction maintenance

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment