Principle:Apache Flink File Commit Finalization
| Knowledge Sources | |
|---|---|
| Domains | Stream_Processing, Fault_Tolerance |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
The commit phase that finalizes pending files by transitioning them from a recoverable pending state to their final committed state, completing the exactly-once delivery guarantee.
Description
File Commit Finalization implements the "commit" phase of the two-phase commit protocol. After a checkpoint succeeds, the committer processes each committable descriptor to finalize the corresponding file operations:
- Pending files: Recovered from their serialized state and committed (typically renamed from a temporary to final location)
- In-progress file cleanup: Stale in-progress files from previous attempts are cleaned up
- Compacted file cleanup: Pre-compaction source files are deleted after compaction
This phase runs idempotently — committing an already-committed file is a no-op — which is essential for exactly-once semantics under failures.
Usage
This principle executes automatically as part of the Flink sink framework after each successful checkpoint. It requires no user configuration but is essential to understand when debugging file visibility issues (files remain invisible until committed).
Theoretical Basis
// Abstract commit algorithm
function commit(committables):
for each committable in committables:
if committable.hasPendingFile():
recover(committable.pendingFile).commitAfterRecovery()
if committable.hasInProgressCleanup():
cleanup(committable.inProgressFile)
if committable.hasCompactedCleanup():
delete(committable.compactedFile)