Heuristic:Heibaiying BigData Notes Kafka Consumer Offset Strategy Tip
| Knowledge Sources | |
|---|---|
| Domains | Messaging, Reliability |
| Last Updated | 2026-02-10 10:00 GMT |
Overview
Use asynchronous offset commit with a callback for production Kafka consumers, and combine async + sync commit on shutdown for reliable exactly-once-like processing.
Description
Kafka offset commit strategy directly affects message delivery guarantees. Auto-commit (default, every 5 seconds) is simple but risks duplicate processing on rebalance. Synchronous commit blocks the consumer thread until the broker acknowledges, reducing throughput. Asynchronous commit with a callback provides the best balance: non-blocking for throughput with error notification for handling failures. The BigData-Notes repository demonstrates a combined pattern: async commit in the main loop with sync commit on shutdown (in a finally block) for maximum reliability.
Usage
Use this heuristic when configuring Kafka consumer offset management for production use. Apply when:
- Default auto-commit causes too many duplicate messages
- Synchronous commit reduces consumer throughput unacceptably
- Implementing at-least-once delivery guarantees
- Handling consumer group rebalancing gracefully
The Insight (Rule of Thumb)
- Action: Set `enable.auto.commit=false`. Use `commitAsync()` with a callback in the poll loop. Call `commitSync()` in the `finally` block for clean shutdown.
- Value: Combine async (high throughput) + sync (guaranteed on shutdown) for optimal reliability.
- Trade-off: Slightly more complex code than auto-commit. Async commit may fail silently without callback handler.
- Bootstrap tip: Always provide at least 2 broker addresses in `bootstrap.servers` for fault tolerance.
Reasoning
Auto-commit at a 5-second interval means any messages consumed but not yet auto-committed will be redelivered after a consumer crash or rebalance. Sync commit guarantees delivery but blocks the consumer. The async+sync combined pattern from the BigData-Notes examples provides the best of both: the main loop uses non-blocking async commits for throughput, while the shutdown handler uses blocking sync commit to ensure the final batch of offsets is committed before the consumer leaves the group. The callback on async commit allows logging and retry logic for transient failures without blocking.