Principle:Apache Flink Hybrid Source Processing
| Knowledge Sources | |
|---|---|
| Domains | Stream_Processing, Source_Architecture |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A delegating read pattern where the hybrid reader forwards polling to the current underlying source reader and signals source completion for switching.
Description
Hybrid Source Processing delegates pollNext to the currently active source reader. When the current reader returns END_OF_INPUT and there are more sources in the chain, the hybrid reader sends a SourceReaderFinishedEvent to the coordinator and returns NOTHING_AVAILABLE while waiting for the switch. For the final source, END_OF_INPUT is propagated directly.
This separation of completion detection (reader-side) from switching coordination (enumerator-side) ensures consistent state across all parallel readers.
Usage
This principle operates automatically during pipeline execution. The user observes a seamless stream of records with no visible source boundaries.
Theoretical Basis
// Abstract processing loop
function pollNext(output):
if currentReader is null:
return NOTHING_AVAILABLE // waiting for switch
status = currentReader.pollNext(output)
if status == END_OF_INPUT AND not isFinalSource:
sendEvent(SourceReaderFinishedEvent(currentSourceIndex))
return NOTHING_AVAILABLE
return status