Heuristic:Fede1024 Rust rdkafka Librdkafka Debug Logging
| Knowledge Sources | |
|---|---|
| Domains | Debugging, Observability |
| Last Updated | 2026-02-07 19:30 GMT |
Overview
Enabling librdkafka's internal debug logs requires both the `RUST_LOG` environment variable and the `debug` configuration option on the producer/consumer; setting only one is insufficient.
Description
rust-rdkafka uses the `log` crate (or optionally `tracing`) to forward librdkafka's internal log messages to the Rust logging framework. However, librdkafka only generates debug-level messages when its `debug` configuration property is set to the appropriate contexts (e.g., `broker,topic,msg`). Without this, even `RUST_LOG=librdkafka=trace` will not produce meaningful output because the C library never emits the messages in the first place. This two-level activation requirement is a common source of confusion.
Usage
Use this heuristic when debugging Kafka connectivity issues, message delivery failures, or consumer rebalancing problems. The combination of `RUST_LOG` and the `debug` config property provides deep visibility into the C library's internal state machine.
The Insight (Rule of Thumb)
- Action: Set both `RUST_LOG="librdkafka=trace,rdkafka::client=debug"` and `.set("debug", "broker,topic,msg")` in your ClientConfig.
- Value: Common debug contexts: `generic`, `broker`, `topic`, `metadata`, `feature`, `queue`, `msg`, `protocol`, `consumer`, `cgrp`, `offset`, `fetch`, `all`.
- Trade-off: Debug logging is extremely verbose and impacts performance. Use only for troubleshooting, never in production.
- Optional: Enable the `tracing` feature for structured tracing events instead of `log` records.
Reasoning
The dual-activation design exists because librdkafka treats logging as a compile-time optimization. The `debug` property tells the C library which subsystems should generate log messages at all. The Rust-side `RUST_LOG` filter then controls which of those messages are actually printed. This separation avoids the overhead of generating log strings that would be filtered out by the Rust logger, but it means you must configure both levels.
Code Evidence
README debugging instructions from `README.md:222-239`:
rust-rdkafka uses the `log` crate to handle logging.
In test and examples, rust-rdkafka uses the `env_logger` crate to format logs.
Logging can be enabled using the `RUST_LOG` environment variable:
RUST_LOG="librdkafka=trace,rdkafka::client=debug" cargo test
To actually receive logs from librdkafka, you also have to set the `debug`
option in the producer or consumer configuration (see librdkafka
[CONFIGURATION.md]).
Main queue polling interval for log events from `src/consumer/mod.rs:116-131`:
/// Returns the minimum interval at which to poll the main queue, which
/// services the logging, stats, and error events.
///
/// By default, the minimum poll interval for the main queue is 1s.
fn main_queue_min_poll_interval(&self) -> Timeout {
Timeout::After(Duration::from_secs(1))
}