Implementation:Apache Flink CompactorOperator NotifyCheckpointComplete

Knowledge Sources	Apache Flink
Domains	Stream_Processing, Fault_Tolerance
Last Updated	2026-02-09 00:00 GMT

Overview

Concrete tool for emitting compacted committables aligned with checkpoint completion provided by the Apache Flink connector-files module.

Description

CompactorOperator.notifyCheckpointComplete resolves completed compaction futures for all requests tracked up to the completed checkpoint ID. For each completed request, it emits CommittableSummary (metadata about the number of committables) followed by CommittableWithLineage<FileSinkCommittable> for each compacted and passthrough committable. State is persisted via RemainingRequestsSerializer (inner class at L279-350) to handle in-progress compactions across failures.

Usage

This is an internal callback invoked by the Flink runtime after checkpoint completion.

Code Reference

Source Location

• Repository: Apache Flink
• File: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
• Lines: L143-208

Signature

@Override
public void endInput() throws Exception {
    // Submit remaining requests and wait for completion
}

@Override
public void notifyCheckpointComplete(long checkpointId) throws Exception {
    // Resolve completed futures and emit committables
}

@Override
public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
    // Track current requests for this checkpoint
}

@Override
public void snapshotState(StateSnapshotContext context) throws Exception {
    // Persist in-progress requests
}

@Override
public void initializeState(StateInitializationContext context) throws Exception {
    // Restore in-progress requests and re-submit
}

Import

import org.apache.flink.connector.file.sink.compactor.operator.CompactorOperator;
// Internal class

I/O Contract

Inputs

Name	Type	Required	Description
checkpointId	long	Yes	Completed checkpoint ID

Outputs

Name	Type	Description
summary	CommittableSummary	Metadata about number of committables
committables	CommittableWithLineage<FileSinkCommittable>	Compacted and passthrough committables

Usage Examples

Emission Flow

// After checkpoint N completes:
// 1. notifyCheckpointComplete(N):
//    a. Get all requests tracked for checkpoints <= N
//    b. For each request: resolve CompletableFuture
//    c. Emit CommittableSummary(numberOfCommittables)
//    d. For each committable in results:
//       emit CommittableWithLineage(committable, checkpointId, subtaskId)
//    e. Clear processed requests
//
// On failure recovery:
// 1. initializeState(): restore serialized requests
// 2. Re-submit restored requests to CompactService
// 3. Resume normal processing

Related Pages

Implements Principle

• Principle:Apache_Flink_Compacted_Committable_Emission