Principle:Apache Flink Batch Formation and Submission

Knowledge Sources	Apache Flink Async Sink Apache Flink
Domains	Stream_Processing, Async_IO
Last Updated	2026-02-09 00:00 GMT

Overview

A batch formation mechanism that extracts entries from the buffer into sized batches and submits them asynchronously to the destination system with result handling.

Description

Batch Formation and Submission bridges the buffering layer and the actual I/O. The BatchCreator extracts entries from the buffer up to the rate limiters allowed batch size, respecting both record count and byte size limits. The resulting Batch is passed to the user-implemented submitRequestEntries method for asynchronous I/O.

Results are handled through a ResultHandler callback:

• complete(): Success — release in-flight slot
• completeExceptionally(Exception): Fatal failure — propagate error
• retryForEntries(List): Partial failure — re-queue failed entries with priority

Usage

Implement submitRequestEntries in your custom async sink to perform the actual I/O (e.g., calling the Kinesis PutRecords API). Use the ResultHandler to report success, failure, or partial success.

Theoretical Basis

// Abstract batch submission algorithm
function flush():
    while buffer.isNotEmpty AND rateLimiter.allowsMore:
        batchSize = rateLimiter.getMaxBatchSize()
        batch = batchCreator.createNextBatch(requestInfo(batchSize), buffer)
        inFlightRequests++
        submitRequestEntries(batch.entries, resultHandler)

function onResult(result):
    if result.isSuccess:
        rateLimiter.registerCompleted(success)
        inFlightRequests--
    else if result.isRetryable:
        buffer.addWithPriority(result.failedEntries)
    else:
        propagateError(result.exception)

Related Pages

Implemented By

• Implementation:Apache_Flink_BatchCreator_CreateNextBatch