Heuristic:Fede1024 Rust rdkafka Manual Offset Store Pattern
| Knowledge Sources | |
|---|---|
| Domains | Messaging, Reliability |
| Last Updated | 2026-02-07 19:30 GMT |
Overview
Decoupling auto-commit from auto-offset-store by setting `enable.auto.offset.store=false` enables at-least-once delivery by controlling exactly which offsets are committed.
Description
By default, librdkafka automatically stores the offset of every received message and periodically auto-commits stored offsets. This means offsets are committed before processing completes, leading to at-most-once semantics (messages may be lost if processing fails). The solution is to disable the automatic offset store while keeping auto-commit enabled. This way, you manually call `store_offset_from_message()` only after successful processing, and the periodic auto-commit handles the actual commit to Kafka.
Usage
Use this heuristic when implementing at-least-once delivery or any pattern where you need processing guarantees. This is the standard approach for consumer applications that cannot afford to lose messages. It avoids the complexity of fully manual commit management while still providing processing-dependent offset control.
The Insight (Rule of Thumb)
- Action: Set `enable.auto.commit=true`, `enable.auto.offset.store=false`, and call `consumer.store_offset_from_message(&msg)` after successful processing.
- Value: `auto.commit.interval.ms=5000` (default) controls how often stored offsets are committed.
- Trade-off: On consumer restart, messages since the last auto-commit will be redelivered (at-least-once, not exactly-once). Processing must be idempotent.
Reasoning
The two-level design (offset store vs. offset commit) is non-obvious but powerful. The offset store is a local in-memory buffer; the commit is the actual write to the Kafka `__consumer_offsets` topic. By controlling the store while letting the library handle commits, you get the convenience of auto-commit with the safety of manual offset management. The alternative (fully manual commits with `enable.auto.commit=false`) requires explicit `commit_message()` calls and careful error handling.
Code Evidence
At-least-once consumer configuration from `examples/at_least_once.rs:52-64`:
let consumer: StreamConsumer<CustomContext> = ClientConfig::new()
.set("group.id", &args.group_id)
.set("bootstrap.servers", &args.brokers)
.set("enable.auto.commit", "true")
.set("auto.commit.interval.ms", "5000")
// but only commit the offsets explicitly stored via
// `consumer.store_offset`.
.set("enable.auto.offset.store", "false")
.set("auto.offset.reset", "earliest")
.create_with_context(context)
.expect("Consumer creation failed");