Principle:Apache Flink Request Buffering
| Knowledge Sources | |
|---|---|
| Domains | Stream_Processing, Buffering |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A buffering mechanism that accumulates converted request entries and triggers non-blocking flushes based on size, count, or time thresholds.
Description
Request Buffering sits between element conversion and batch formation. Each converted request entry is wrapped with size metadata and added to a buffer (backed by an ArrayDeque). The buffer tracks total size in bytes and triggers automatic flushes when thresholds are reached:
- Size threshold: Total buffered bytes exceed the flush threshold
- Count threshold: Number of buffered entries exceeds the max batch size
- Time threshold: Elements have been buffered longer than maxTimeInBufferMS
Flushing is non-blocking — it uses Flinks MailboxExecutor to schedule the flush in the operators mailbox, ensuring thread safety without locks.
Usage
This principle operates internally within the AsyncSinkWriter. Users configure its behavior indirectly through the builder parameters (max batch size, max buffered requests, max time in buffer).
Theoretical Basis
// Abstract buffering algorithm
function write(element, context):
entry = elementConverter.apply(element, context)
wrapper = new RequestEntryWrapper(entry, getSizeInBytes(entry))
buffer.add(wrapper)
if buffer.size >= maxBatchSize OR buffer.totalBytes >= flushThreshold:
scheduleFlush()
if not timerActive AND maxTimeInBufferMS > 0:
scheduleTimerFlush(maxTimeInBufferMS)