Principle:Apache Flink Split Based Record Reading
| Knowledge Sources | |
|---|---|
| Domains | Stream_Processing, File_IO |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A split-based reading pattern where a source reader processes assigned file splits by creating format-specific readers, fetching record batches, and emitting individual records downstream.
Description
Split Based Record Reading implements the FLIP-27 source reader contract for file-based sources. The architecture uses a single-threaded multiplexed reader that processes splits sequentially within a single thread. For each split, a BulkFormat.Reader is created to read batches of records (as RecordIterator<T>). Records are wrapped in RecordAndPosition<T> to track offset within the split for checkpointing. When a split is finished, the reader requests the next split from the enumerator.
The design multiplexes multiple splits through a single reader thread, avoiding the complexity of multi-threaded file I/O while maintaining efficient I/O through batch-oriented reading.
Usage
This principle operates as the runtime execution of file reading. Users configure it indirectly through the FileSource builder by selecting the format (stream or bulk). Stream formats are internally adapted to bulk format via StreamFormatAdapter.
Theoretical Basis
// Abstract algorithm
function readSplits():
while splits available:
reader = format.createReader(currentSplit)
while batch = reader.readBatch():
for each record in batch:
emit(record, position)
requestNextSplit()