Principle:Apache Flink Compacted Committable Emission
| Knowledge Sources | |
|---|---|
| Domains | Stream_Processing, Fault_Tolerance |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A checkpoint-aligned emission mechanism that forwards compacted file committables to the downstream committer after verifying compaction completion.
Description
Compacted Committable Emission ensures that compacted files are only committed after the compaction is verified complete. The operator tracks compaction requests per checkpoint in a TreeMap<Long, List<CompactorRequest>>. On notifyCheckpointComplete, completed compaction futures are resolved and their results emitted as CommittableMessage<FileSinkCommittable> (including CommittableSummary and CommittableWithLineage).
The emitted committables include:
- New pending files from compacted output
- Cleanup entries for the original pre-compaction source files
- Passthrough entries from the original request
State persistence uses RemainingRequestsSerializer to checkpoint in-progress requests across failures.
Usage
This principle is internal to the compaction pipeline and operates automatically aligned with Flink checkpoints.
Theoretical Basis
// Abstract emission flow
function notifyCheckpointComplete(checkpointId):
completedRequests = checkpointRequests.headMap(checkpointId + 1)
for each request in completedRequests:
results = request.future.get()
emit(CommittableSummary(count))
for each committable in results:
emit(CommittableWithLineage(committable))
completedRequests.clear()