Implementation:Apache Flink Pool
| Knowledge Sources | |
|---|---|
| Domains | Connectors, File_Connector |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A generic object pool that caches and recycles heavyweight objects to reduce allocation overhead in file source readers.
Description
Pool is a thread-safe generic object pool backed by an ArrayBlockingQueue. It is designed for use within BulkFormat.Reader implementations where returned objects are heavyweight and benefit from reuse. The pool separates the concern of object creation from recycling: objects are added to the pool up to a fixed capacity, and a Recycler functional interface is provided to return objects back to the pool after use. Because file reading occurs in I/O threads while record processing occurs in Flink's main processing threads, objects cannot be reused immediately; they must first be recycled back through the pool. The class supports both blocking retrieval via pollEntry() and non-blocking retrieval via tryPollEntry().
The inner Recycler<T> functional interface encapsulates the callback mechanism for returning objects to their owning pool. It is automatically wired to the pool's internal addBack method via a method reference.
Usage
Use Pool when implementing a BulkFormat.Reader that produces heavyweight result objects (such as large record batches or column vectors) that should be recycled rather than garbage collected. Create the pool with a desired capacity, add initial objects with add(), obtain the recycler() to attach to produced objects, retrieve entries with pollEntry() or tryPollEntry(), and rely on consumers to recycle entries back after processing.
Code Reference
Source Location
- Repository: Apache_Flink
- File: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/util/Pool.java
- Lines: 1-109
Signature
@PublicEvolving
public class Pool<T> {
public Pool(int poolCapacity);
public Recycler<T> recycler();
public synchronized void add(T object);
public T pollEntry() throws InterruptedException;
@Nullable public T tryPollEntry();
void addBack(T object);
@FunctionalInterface
public interface Recycler<T> {
void recycle(T object);
}
}
Import
import org.apache.flink.connector.file.src.util.Pool;
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| poolCapacity | int | Yes | Maximum number of objects the pool can hold; set at construction time. |
| object | T | Yes | An object to add to the pool via add() or to recycle via the Recycler. |
Outputs
| Name | Type | Description |
|---|---|---|
| recycler | Recycler<T> | A recycler instance bound to this pool, used to return objects after processing. |
| pollEntry result | T | The next available pooled object (blocking). Returns from the internal queue. |
| tryPollEntry result | T (nullable) | The next available pooled object or null if the pool is currently empty. |
Usage Examples
// Create a pool with capacity for 2 record batch objects
Pool<RecordBatch> pool = new Pool<>(2);
// Add initial objects
pool.add(new RecordBatch());
pool.add(new RecordBatch());
// Obtain the recycler to attach to produced results
Pool.Recycler<RecordBatch> recycler = pool.recycler();
// In the reader thread: get an available batch (blocks if none available)
RecordBatch batch = pool.pollEntry();
// ... fill the batch with records and return it to the downstream operator ...
// In the processing thread: after processing, recycle the batch back
recycler.recycle(batch);