Implementation:Apache Paimon ExpireConfig
| Knowledge Sources | |
|---|---|
| Domains | Configuration, Data Retention |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
ExpireConfig encapsulates retention policies for snapshots and changelogs in Apache Paimon tables.
Description
ExpireConfig is a comprehensive configuration class that governs data retention policies in Paimon. It manages expiration settings for both snapshots (which represent the full state of a table at specific points in time) and changelogs (which capture incremental changes between snapshots). The configuration supports both time-based and count-based retention rules.
The class maintains separate but parallel configuration for snapshots and changelogs, each with four parameters: maximum number to retain (RetainMax), minimum number to retain (RetainMin), time duration to retain (TimeRetain), and maximum number of deletions per expiration run (MaxDeletes). The minimum constraints ensure that some history is always preserved even when time-based expiration would remove everything, while maximum deletion limits prevent overwhelming deletion operations.
A key feature is changelog decoupling detection, which automatically determines whether changelog retention is configured independently from snapshot retention. When changelog retention exceeds snapshot retention in any dimension (count or time), the `changelogDecoupled` flag is set to true, indicating that changelogs should be managed separately from their associated snapshots.
The class uses a builder pattern for construction, providing sensible defaults (Integer.MAX_VALUE for max counts, 1 for min counts, Long.MAX_VALUE milliseconds for time retention) that effectively disable expiration unless explicitly configured. The builder allows selective override of snapshot settings, with changelog settings inheriting from snapshot settings when not explicitly specified.
Usage
Use ExpireConfig when configuring table properties related to snapshot and changelog retention. This configuration is typically parsed from table options and passed to snapshot management components that perform periodic cleanup. The builder pattern allows flexible configuration where only the needed retention parameters are specified, with the rest using sensible defaults.
Code Reference
Source Location
- Repository: Apache_Paimon
- File: paimon-api/src/main/java/org/apache/paimon/options/ExpireConfig.java
Signature
public class ExpireConfig {
private final int snapshotRetainMax;
private final int snapshotRetainMin;
private final Duration snapshotTimeRetain;
private final int snapshotMaxDeletes;
private final int changelogRetainMax;
private final int changelogRetainMin;
private final Duration changelogTimeRetain;
private final int changelogMaxDeletes;
private final boolean changelogDecoupled;
public ExpireConfig(
int snapshotRetainMax,
int snapshotRetainMin,
Duration snapshotTimeRetain,
int snapshotMaxDeletes,
int changelogRetainMax,
int changelogRetainMin,
Duration changelogTimeRetain,
int changelogMaxDeletes)
public int getSnapshotRetainMax()
public int getSnapshotRetainMin()
public Duration getSnapshotTimeRetain()
public int getSnapshotMaxDeletes()
public int getChangelogRetainMax()
public int getChangelogRetainMin()
public Duration getChangelogTimeRetain()
public int getChangelogMaxDeletes()
public boolean isChangelogDecoupled()
public static Builder builder()
public static final class Builder {
public Builder snapshotRetainMax(int snapshotRetainMax)
public Builder snapshotRetainMin(int snapshotRetainMin)
public Builder snapshotTimeRetain(Duration snapshotTimeRetain)
public Builder snapshotMaxDeletes(int snapshotMaxDeletes)
public Builder changelogRetainMax(Integer changelogRetainMax)
public Builder changelogRetainMin(Integer changelogRetainMin)
public Builder changelogTimeRetain(Duration changelogTimeRetain)
public Builder changelogMaxDeletes(Integer changelogMaxDeletes)
public ExpireConfig build()
}
}
Import
import org.apache.paimon.options.ExpireConfig;
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| snapshotRetainMax | int | No | Maximum snapshots to retain (default: Integer.MAX_VALUE) |
| snapshotRetainMin | int | No | Minimum snapshots to retain (default: 1) |
| snapshotTimeRetain | Duration | No | Time duration to retain snapshots (default: Long.MAX_VALUE ms) |
| snapshotMaxDeletes | int | No | Max deletions per run (default: Integer.MAX_VALUE) |
| changelogRetainMax | Integer | No | Maximum changelogs to retain (inherits from snapshot if null) |
| changelogRetainMin | Integer | No | Minimum changelogs to retain (inherits from snapshot if null) |
| changelogTimeRetain | Duration | No | Time to retain changelogs (inherits from snapshot if null) |
| changelogMaxDeletes | Integer | No | Max changelog deletions per run (inherits from snapshot if null) |
Outputs
| Name | Type | Description |
|---|---|---|
| expireConfig | ExpireConfig | Immutable configuration instance |
| snapshotRetainMax | int | Maximum snapshot retention count |
| snapshotRetainMin | int | Minimum snapshot retention count |
| snapshotTimeRetain | Duration | Snapshot time retention |
| snapshotMaxDeletes | int | Maximum snapshot deletions per run |
| changelogRetainMax | int | Maximum changelog retention count |
| changelogRetainMin | int | Minimum changelog retention count |
| changelogTimeRetain | Duration | Changelog time retention |
| changelogMaxDeletes | int | Maximum changelog deletions per run |
| changelogDecoupled | boolean | Whether changelog retention is independent |
Usage Examples
// Basic snapshot retention - keep last 10 snapshots
ExpireConfig config1 = ExpireConfig.builder()
.snapshotRetainMax(10)
.snapshotRetainMin(1)
.build();
// Time-based retention - keep 7 days of snapshots
ExpireConfig config2 = ExpireConfig.builder()
.snapshotTimeRetain(Duration.ofDays(7))
.snapshotRetainMin(3) // but always keep at least 3
.build();
// Combined count and time retention
ExpireConfig config3 = ExpireConfig.builder()
.snapshotRetainMax(100)
.snapshotRetainMin(10)
.snapshotTimeRetain(Duration.ofDays(30))
.build();
// Limiting deletion rate
ExpireConfig config4 = ExpireConfig.builder()
.snapshotRetainMax(50)
.snapshotMaxDeletes(10) // delete max 10 per run
.build();
// Decoupled changelog retention
ExpireConfig config5 = ExpireConfig.builder()
.snapshotRetainMax(20)
.snapshotTimeRetain(Duration.ofDays(7))
.changelogRetainMax(100) // keep more changelogs
.changelogTimeRetain(Duration.ofDays(30)) // for longer time
.build();
System.out.println(config5.isChangelogDecoupled()); // true
// Changelog inherits from snapshot when not specified
ExpireConfig config6 = ExpireConfig.builder()
.snapshotRetainMax(50)
.snapshotTimeRetain(Duration.ofDays(14))
.build();
// changelog settings will match snapshot settings
assert config6.getChangelogRetainMax() == 50;
assert config6.getChangelogTimeRetain().equals(Duration.ofDays(14));
assert !config6.isChangelogDecoupled();
// Using in table configuration
Map<String, String> tableOptions = new HashMap<>();
tableOptions.put("snapshot.num-retained.max", "100");
tableOptions.put("snapshot.num-retained.min", "10");
tableOptions.put("snapshot.time-retained", "7 d");
ExpireConfig expireConfig = ExpireConfig.builder()
.snapshotRetainMax(
Integer.parseInt(tableOptions.get("snapshot.num-retained.max"))
)
.snapshotRetainMin(
Integer.parseInt(tableOptions.get("snapshot.num-retained.min"))
)
.snapshotTimeRetain(
parseDuration(tableOptions.get("snapshot.time-retained"))
)
.build();
// Using in snapshot expiration logic
public void expireSnapshots(List<Snapshot> snapshots, ExpireConfig config) {
long cutoffTime = System.currentTimeMillis()
- config.getSnapshotTimeRetain().toMillis();
List<Snapshot> toDelete = snapshots.stream()
.filter(s -> s.timeMillis() < cutoffTime)
.skip(config.getSnapshotRetainMin()) // always keep min
.limit(config.getSnapshotMaxDeletes()) // respect max deletes
.collect(Collectors.toList());
if (snapshots.size() - toDelete.size() > config.getSnapshotRetainMax()) {
// Also need to remove excess beyond max
}
deleteSnapshots(toDelete);
}