Principle:Apache Flink Element Conversion
| Knowledge Sources | |
|---|---|
| Domains | Stream_Processing, Data_Transformation |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A transformation interface that converts pipeline elements into destination-specific request entries before buffering and batching.
Description
Element Conversion decouples the Flink pipelines data model from the external systems request format. Each incoming record is transformed into a serializable request entry that the destination system can accept (e.g., a Kinesis PutRecordRequest, a DynamoDB WriteRequest). This separation enables the async sink framework to handle buffering, batching, and retries generically, regardless of the destination format.
The conversion happens synchronously in the write path, before the element enters the buffer. The converter also receives a SinkWriter.Context providing the elements timestamp and watermark for use in conversion logic.
Usage
Implement this interface when building a custom async sink connector. The conversion function should be lightweight (no I/O) as it runs in the critical write path. Heavy transformations should be performed upstream in the pipeline.
Theoretical Basis
// Abstract conversion pattern
function convert(element, context):
requestEntry = new RequestEntry()
requestEntry.payload = serialize(element)
requestEntry.timestamp = context.timestamp()
requestEntry.key = extractKey(element)
return requestEntry // Serializable, destination-specific