Heuristic:Fede1024 Rust rdkafka Regular Polling Required
| Knowledge Sources | |
|---|---|
| Domains | Messaging, Optimization |
| Last Updated | 2026-02-07 19:30 GMT |
Overview
The BaseProducer requires regular `poll()` calls to process delivery callbacks; failure to poll frequently enough causes `QueueFull` errors that block message production.
Description
librdkafka's producer is fully asynchronous: it buffers messages internally and sends them in batches. However, delivery results (success/failure) are queued as events that must be consumed by calling `poll()`. If the event queue is not drained, it fills up and subsequent `send()` calls fail with `QueueFull`. This is a fundamental design constraint of the C library that surprises developers who expect a fully fire-and-forget async API.
Usage
Use this heuristic when designing a producer application with `BaseProducer`. If you see `QueueFull` errors during production, the root cause is almost always insufficient polling. Consider switching to `ThreadedProducer` (which auto-polls at 100ms intervals) or `FutureProducer` (which handles polling internally via the async runtime) to avoid this issue entirely.
The Insight (Rule of Thumb)
- Action: Call `BaseProducer::poll()` at regular intervals in a dedicated loop or thread.
- Value: `ThreadedProducer` polls every 100ms by default; this is a good baseline.
- Trade-off: More frequent polling reduces latency for delivery callbacks but increases CPU usage. Less frequent polling saves CPU but risks `QueueFull` under high throughput.
- Alternative: Use `ThreadedProducer` (spawns a dedicated polling thread) or `FutureProducer` (integrates with async runtime) to avoid manual polling.
Reasoning
The C librdkafka library uses a single internal event queue for delivery notifications. When a message is successfully delivered or permanently fails, the result is enqueued. The `poll()` call dequeues these events and executes the user's delivery callback on the calling thread. If delivery events accumulate faster than they are consumed, the internal buffer fills and new messages are rejected with `QueueFull`. The `ThreadedProducer` solves this by running `poll(Duration::from_millis(100))` in a loop on a dedicated thread, and the `FutureProducer` offloads polling to the async runtime.
Code Evidence
Documentation from `src/producer/base_producer.rs:26-42`:
//! ### Calling poll
//!
//! To execute delivery callbacks the `poll` method of the producer should be
//! called regularly. If `poll` is not called, or not often enough, a
//! [`RDKafkaErrorCode::QueueFull`] error will be returned.
//!
//! ## `ThreadedProducer`
//!
//! The `ThreadedProducer` is a wrapper around the `BaseProducer` which spawns a
//! thread dedicated to calling `poll` on the producer at regular intervals
ThreadedProducer polling loop from `src/producer/base_producer.rs:687-696`:
thread::Builder::new()
.name("producer polling thread".to_string())
.spawn(move || {
trace!("Polling thread loop started");
loop {
producer.poll(Duration::from_millis(100));
if should_stop.load(Ordering::Relaxed) {
break;
}
}
})