Principle:Fede1024 Rust rdkafka Transactional Offset Commit
| Knowledge Sources | |
|---|---|
| Domains | Messaging, Transactions, Reliability |
| Last Updated | 2026-02-07 19:00 GMT |
Overview
A mechanism for atomically committing consumer offsets together with produced messages within a Kafka transaction.
Description
Transactional Offset Commit ensures that consumer offset advancement and message production are atomic. Without this, a failure between producing output and committing the input offset would cause either duplicate processing (offset not committed) or data loss (offset committed but output not produced).
By sending consumer offsets to the transaction coordinator as part of the transaction, both the produced messages and the offset advancement are committed (or aborted) together. This is the core mechanism enabling exactly-once semantics in consume-transform-produce pipelines.
The consumer must be configured with enable.auto.commit=false so that offsets are only committed through the transaction.
Usage
Use this principle in exactly-once consume-transform-produce pipelines. After producing output messages within a transaction, call send_offsets_to_transaction with the consumed offsets and consumer group metadata before committing the transaction.
Theoretical Basis
Pseudo-code logic:
// Abstract algorithm
consumer_offsets = build_offset_list(consumed_messages)
group_metadata = consumer.group_metadata()
producer.begin_transaction()
producer.send(output_records)
producer.send_offsets_to_transaction(consumer_offsets, group_metadata, timeout)
producer.commit_transaction(timeout)
// Both output messages AND input offsets committed atomically