Heuristic:Apache Dolphinscheduler Failover Marker TTL
| Knowledge Sources | |
|---|---|
| Domains | Fault_Tolerance, Distributed_Systems |
| Last Updated | 2026-02-10 10:00 GMT |
Overview
Failover finished markers in the registry have a 7-day TTL and are cleaned up automatically to prevent unbounded registry growth.
Description
When DolphinScheduler completes a Master or Worker failover, it persists a failover finished marker in the registry (ZooKeeper) under the `FAILOVER_FINISH_NODES` path. The `cleanHistoryFailoverFinishedMarks()` method periodically scans these markers and removes any that are older than 7 days. This prevents the registry from growing unbounded with historical failover records. Additionally, failover operations are idempotent: before initiating a failover, the system checks whether a marker with the same address and deadline timestamp already exists, skipping duplicate failovers.
Usage
Apply this heuristic when monitoring registry health or debugging failover behavior. If the registry has many entries under the failover path, they will auto-clean after 7 days. If failovers appear to be skipped, check for existing markers that indicate the failover already completed.
The Insight (Rule of Thumb)
- Action: No manual intervention needed. Failover markers auto-expire after 7 days.
- Value: TTL = 7 days (`TimeUnit.DAYS.toMillis(7)`).
- Trade-off: 7 days allows sufficient time for debugging failed failovers before markers are removed. Shorter TTL would save registry space but lose debugging information sooner.
- Idempotency: Duplicate failover requests are detected by checking `path + deadline timestamp` in the registry before proceeding.
Reasoning
Without TTL-based cleanup, every failover event would leave a permanent marker in ZooKeeper. Over time, this accumulates and degrades ZooKeeper performance (watch overhead, snapshot size). The 7-day window balances operational debugging needs (you can still inspect recent failovers) with registry hygiene. The idempotency check prevents race conditions where multiple Masters might try to failover the same dead node simultaneously.
Code evidence from `FailoverCoordinator.java:216-238`:
public void cleanHistoryFailoverFinishedMarks() {
// Clean the history failover finished nodes
// which failover is before the current time minus 1 week
final Collection<String> failoverFinishedMarkSuffixes =
registryClient.getChildrenKeys(RegistryNodeType.FAILOVER_FINISH_NODES.getRegistryPath());
if (CollectionUtils.isEmpty(failoverFinishedMarkSuffixes)) {
return;
}
for (final String failoverFinishedMarkSuffix : failoverFinishedMarkSuffixes) {
final String failoverFinishedMarkFullPath = RegistryNodeType.FAILOVER_FINISH_NODES.getRegistryPath()
+ Constants.SINGLE_SLASH + failoverFinishedMarkSuffix;
try {
final String failoverFinishTime = registryClient.get(failoverFinishedMarkFullPath);
if (System.currentTimeMillis() - Long.parseLong(failoverFinishTime) > TimeUnit.DAYS.toMillis(7)) {
registryClient.remove(failoverFinishedMarkFullPath);
log.info(
"Clear the failover finished node: {} which failover time is before the current time minus 1 week",
failoverFinishedMarkFullPath);
}
} catch (Exception ex) {
log.error("Failed to clean the failoverFinishedNode: {}", failoverFinishedMarkFullPath, ex);
}
}
}