Implementation:Apache Flink ArrayResultIterator
| Knowledge Sources | |
|---|---|
| Domains | Connectors, Source_Framework |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A mutable, recyclable BulkFormat.RecordIterator implementation that returns elements of an array one at a time, augmented with checkpoint position information.
Description
ArrayResultIterator is a public-evolving implementation of BulkFormat.RecordIterator that iterates over an array of records and produces RecordAndPosition values for each element. It extends RecyclableIterator to support optional resource recycling when the batch is released.
The class is designed for mutability and object reuse:
- The set() method allows reconfiguring the iterator with a new array of records, a count, an offset, and a starting skip count, enabling reuse without allocation.
- Internally, it uses a single MutableRecordAndPosition instance that is updated on each next() call, avoiding per-record object creation.
- The position tracking follows the convention that each record's position points AFTER itself, so the first returned record has a records-to-skip count of skipCountOfFirst + 1.
Key design decisions:
- Mutable design: The iterator can be reset via set() for different arrays, supporting batch processing patterns where the same iterator object is reused across batches.
- Recycler support: An optional Runnable recycler is called when releaseBatch() is invoked, enabling custom cleanup or pool return logic.
- Array-backed: Uses a simple array with index tracking for efficient sequential access.
Usage
Use ArrayResultIterator in BulkFormat reader implementations when records are available as an array and need to be returned with checkpoint position information. It is particularly suitable for formats that read data in array-based batches (e.g., columnar formats) and need to expose records through the RecordIterator interface.
Code Reference
Source Location
- Repository: Apache_Flink
- File:
flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/util/ArrayResultIterator.java - Lines: 1-89
Signature
@PublicEvolving
public final class ArrayResultIterator<E> extends RecyclableIterator<E>
implements BulkFormat.RecordIterator<E>
Import
import org.apache.flink.connector.file.src.util.ArrayResultIterator;
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| recycler | Runnable | No | An optional callback invoked when releaseBatch() is called (null for no-op) |
| records | E[] | Yes (via set()) | The array of records to iterate over |
| num | int | Yes (via set()) | The number of valid records in the array to return |
| offset | long | Yes (via set()) | The checkpoint offset to associate with all records in this batch |
| skipCountOfFirst | long | Yes (via set()) | The base skip count; the first record gets skipCountOfFirst + 1 |
Outputs
| Name | Type | Description |
|---|---|---|
| next() | RecordAndPosition<E> | The next record paired with its checkpoint position, or null when all records have been returned |
Usage Examples
// Create an ArrayResultIterator and populate it with records
ArrayResultIterator<String> iterator = new ArrayResultIterator<>();
String[] batch = {"record1", "record2", "record3"};
long fileOffset = 0L;
long skipCount = 0L;
// Set the batch of records with position information
iterator.set(batch, batch.length, fileOffset, skipCount);
// Iterate through records
RecordAndPosition<String> recordAndPos;
while ((recordAndPos = iterator.next()) != null) {
String record = recordAndPos.getRecord();
// process record...
}
// Release the batch (triggers recycler if provided)
iterator.releaseBatch();