Heuristic:Fede1024 Rust rdkafka Main Queue Poll Interval
| Knowledge Sources | |
|---|---|
| Domains | Messaging, Observability |
| Last Updated | 2026-02-07 19:30 GMT |
Overview
The consumer's main queue (handling stats, logs, and errors) is polled at a minimum interval of 1 second by default, configurable via `ConsumerContext::main_queue_min_poll_interval()`.
Description
librdkafka maintains two queues: the consumer queue (messages) and the main queue (stats, error events, log events). When `BaseConsumer::poll()` is called with a timeout longer than the main queue interval, the implementation polls the main queue at the configured interval while waiting for messages. This ensures that statistics callbacks, error events, and log messages are processed even during periods of no message activity. The default 1-second interval balances responsiveness with overhead.
Usage
Use this heuristic when tuning statistics collection frequency or debugging missing stats/error callbacks. If `statistics.interval.ms` is set but callbacks seem delayed, the main queue poll interval may be the bottleneck. Override `main_queue_min_poll_interval()` in your `ConsumerContext` to reduce the interval for more responsive event handling.
The Insight (Rule of Thumb)
- Action: Override `ConsumerContext::main_queue_min_poll_interval()` to return a shorter duration if you need faster stats/error event processing.
- Value: Default: 1 second. For high-frequency stats: 200ms. For very responsive error handling: 100ms.
- Trade-off: Shorter intervals increase CPU usage from more frequent polling. The 1-second default is appropriate for most applications.
Reasoning
Without main queue polling, librdkafka events would only be processed when messages arrive on the consumer queue. During idle periods (no messages), stats callbacks would stall, error events would be delayed, and log messages would not appear. The periodic main queue poll ensures these events flow regardless of message activity.
Code Evidence
Main queue poll interval from `src/consumer/mod.rs:116-132`:
/// Returns the minimum interval at which to poll the main queue, which
/// services the logging, stats, and error events.
///
/// The main queue is polled once whenever [`BaseConsumer::poll`] is called.
/// If `poll` is called with a timeout that is larger than this interval,
/// then the main queue will be polled at that interval while the consumer
/// queue is blocked.
///
/// For example, if the main queue's minimum poll interval is 200ms and
/// `poll` is called with a timeout of 1s, then `poll` may block for up to
/// 1s waiting for a message, but it will poll the main queue every 200ms
/// while it is waiting.
///
/// By default, the minimum poll interval for the main queue is 1s.
fn main_queue_min_poll_interval(&self) -> Timeout {
Timeout::After(Duration::from_secs(1))
}