Principle:Fede1024 Rust rdkafka Manual Offset Management
| Knowledge Sources | |
|---|---|
| Domains | Messaging, Reliability |
| Last Updated | 2026-02-07 19:00 GMT |
Overview
A technique for storing consumer offsets only after successful message processing, providing at-least-once delivery guarantees.
Description
Manual Offset Management disables automatic offset storage (enable.auto.offset.store=false) so that the consumer does not mark messages as consumed until the application explicitly stores the offset. This ensures that if processing fails or the consumer crashes, the message will be re-delivered upon restart.
The pattern is: consume a message, process it (potentially producing to another topic), and only then store the offset. The stored offset is picked up by the next auto-commit cycle (auto.commit.interval.ms) or explicit commit call. This provides at-least-once semantics: messages may be processed more than once on failure, but never lost.
Usage
Use this principle when you need at-least-once delivery guarantees. It is essential for consume-transform-produce pipelines where you want to ensure the input message offset is only advanced after the output message is successfully produced.
Theoretical Basis
At-least-once delivery requires separating offset advancement from message receipt:
Pseudo-code logic:
// Abstract algorithm
// Config: enable.auto.offset.store = false
message = consumer.recv()
result = process_and_produce(message)
if result.is_ok() {
consumer.store_offset_from_message(message) // only store on success
}
// Auto-commit picks up stored offset later
The guarantee: if processing fails, the offset is never stored, so the message will be re-consumed on the next poll.