Principle:Apache Flink Cross Source Checkpointing

Knowledge Sources	Apache Flink Checkpointing Apache Flink
Domains	Fault_Tolerance, Source_Architecture
Last Updated	2026-02-09 00:00 GMT

Overview

A checkpoint mechanism that persists the current source index and wrapped source-specific state to enable recovery at the correct position within a multi-source chain.

Description

Cross Source Checkpointing solves the problem of checkpointing state across heterogeneous sources. The HybridSourceEnumeratorState wraps the current source index and the active source enumerators serialized state. HybridSourceSplit wraps source-specific splits with the source index and serialized split bytes.

On recovery:

• The enumerator restores at the checkpointed source index with the underlying enumerator state
• Readers restore their splits and determine which source they belong to from the HybridSourceSplit.sourceIndex
• The SwitchedSources registry maps source indices to their serializers for deserialization

Usage

This principle operates transparently when checkpointing is enabled. It ensures that after a failure, the pipeline resumes from the correct source and position in the chain.

Theoretical Basis

// Abstract checkpointing
function snapshotEnumeratorState(checkpointId):
    wrappedState = currentEnumerator.snapshotState(checkpointId)
    return HybridSourceEnumeratorState(
        currentSourceIndex,
        serialize(wrappedState),
        serializerVersion)

function restoreEnumerator(state):
    sourceIndex = state.currentSourceIndex
    wrappedState = deserialize(state.wrappedStateBytes)
    createEnumeratorAtIndex(sourceIndex, wrappedState)

Related Pages

Implemented By

• Implementation:Apache_Flink_HybridSourceEnumeratorState_Snapshot