Implementation:Apache Flink IteratorResultIterator
| Knowledge Sources | |
|---|---|
| Domains | Connectors, Source_Framework |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A BulkFormat.RecordIterator implementation that wraps a standard Java Iterator and augments each returned element with checkpoint position information.
Description
IteratorResultIterator is a public-evolving implementation of BulkFormat.RecordIterator that adapts a Java Iterator<E> into the Flink file source iterator protocol. It extends RecyclableIterator to support optional resource recycling and produces RecordAndPosition values that pair each record with its checkpoint position.
The class is constructed with an iterator of records, an offset value, and a starting skip count. All records share the same offset value, while the skip count is incremented by one for each record returned via next(). This follows the convention that each record's position points AFTER itself -- a checkpoint taken after emitting a record must resume from the position after that record.
Key characteristics:
- Iterator-backed: Wraps any standard Iterator<E>, making it easy to adapt existing collection-based or streaming data sources.
- Single MutableRecordAndPosition: Uses one mutable position object that is updated in-place for each next() call, minimizing object allocation.
- Recycler support: An optional Runnable recycler can be supplied and is invoked when releaseBatch() is called.
- Immutable after construction: Unlike ArrayResultIterator, this class does not support resetting with new data after construction.
Usage
Use IteratorResultIterator in BulkFormat reader implementations when records are available via a Java Iterator and need to be returned with checkpoint position information. It is suitable for formats that produce records lazily through an iterator rather than materializing them into an array.
Code Reference
Source Location
- Repository: Apache_Flink
- File:
flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/util/IteratorResultIterator.java - Lines: 1-94
Signature
@PublicEvolving
public final class IteratorResultIterator<E> extends RecyclableIterator<E>
implements BulkFormat.RecordIterator<E>
Import
import org.apache.flink.connector.file.src.util.IteratorResultIterator;
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| records | Iterator<E> | Yes | The Java iterator providing the records to return |
| offset | long | Yes | The checkpoint offset to associate with all records from this iterator |
| startingSkipCount | long | Yes | The base skip count; the first record gets startingSkipCount + 1 |
| recycler | Runnable | No | An optional callback invoked when releaseBatch() is called (null for no-op) |
Outputs
| Name | Type | Description |
|---|---|---|
| next() | RecordAndPosition<E> | The next record paired with its checkpoint position, or null when the underlying iterator is exhausted |
Usage Examples
// Create an IteratorResultIterator from a list's iterator
List<String> records = Arrays.asList("alpha", "beta", "gamma");
long fileOffset = 1024L;
long startingSkipCount = 0L;
IteratorResultIterator<String> iterator =
new IteratorResultIterator<>(records.iterator(), fileOffset, startingSkipCount);
// Iterate through records with position information
RecordAndPosition<String> recordAndPos;
while ((recordAndPos = iterator.next()) != null) {
String record = recordAndPos.getRecord();
long skipCount = recordAndPos.getRecordSkipCount();
// process record with position info...
}
// Release the batch
iterator.releaseBatch();