Heuristic:Fede1024 Rust rdkafka Queue Buffering Priority
| Knowledge Sources | |
|---|---|
| Domains | Messaging, Optimization |
| Last Updated | 2026-02-07 19:30 GMT |
Overview
The `queue.buffering.max.kbytes` setting has higher priority than `queue.buffering.max.messages`, and `queue.buffering.max.ms=0` disables batching for immediate delivery at the cost of throughput.
Description
librdkafka's producer queue has three buffering controls: message count, total byte size, and linger time. A common mistake is tuning `queue.buffering.max.messages` without realizing that the byte size limit (`queue.buffering.max.kbytes`) takes precedence. If your messages are large, you may hit the byte limit long before the message count limit. Additionally, `queue.buffering.max.ms` controls how long the producer waits to batch messages; setting it to 0 sends immediately but sacrifices batching efficiency and compression effectiveness.
Usage
Use this heuristic when tuning producer throughput or debugging unexpected QueueFull errors. If you see QueueFull despite a high `max.messages` setting, check whether `max.kbytes` is the actual bottleneck. For at-least-once delivery patterns where latency matters more than throughput, set `queue.buffering.max.ms=0`.
The Insight (Rule of Thumb)
- Action: Always consider all three buffering parameters together: `queue.buffering.max.messages` (default: 100000), `queue.buffering.max.kbytes` (default: 1048576), and `queue.buffering.max.ms` (default: 0).
- Value: `max.kbytes` has higher priority than `max.messages`. For low-latency: `max.ms=0`. For high-throughput: increase `max.ms` (e.g., 5-100ms).
- Trade-off: Higher `max.ms` improves batching and compression but increases delivery latency. Setting `max.ms=0` gives immediate sends but increases per-message overhead.
Reasoning
The priority relationship between kbytes and messages is documented but easy to miss. In practice, if you produce 10KB messages and have the default 1GB byte limit, you can buffer ~100K messages. But with 100KB messages, the byte limit kicks in at ~10K messages regardless of `max.messages`. The `max.ms=0` default means rdkafka sends messages as soon as possible, which is safe but suboptimal for throughput-oriented workloads.
Code Evidence
Configuration documentation from `src/producer/mod.rs:129-138`:
//! - `queue.buffering.max.messages`: Maximum number of messages allowed on the
//! producer queue. Default: 100000.
//! - `queue.buffering.max.kbytes`: Maximum total message size sum allowed on
//! the producer queue. This property has higher priority than
//! queue.buffering.max.messages. Default: 1048576.
//! - `queue.buffering.max.ms`: Delay in milliseconds to wait for messages in
//! the producer queue to accumulate before sending a request to the brokers.
//! A higher value allows larger and more effective (less overhead, improved
//! compression) batches of messages to accumulate at the expense of increased
//! message delivery latency. Default: 0.
At-least-once example using `max.ms=0` from `examples/at_least_once.rs:79`:
.set("queue.buffering.max.ms", "0") // Do not buffer