Principle:Fede1024 Rust rdkafka Cluster Metadata Inspection
| Knowledge Sources | |
|---|---|
| Domains | Cluster_Administration, Metadata, Topology_Discovery |
| Last Updated | 2026-02-07 19:00 GMT |
Overview
Principle of querying Kafka cluster topology to discover brokers, topics, partitions, replica assignments, and ISR (in-sync replica) lists for operational monitoring and diagnostic purposes.
Description
Kafka exposes cluster metadata through a dedicated Metadata API request. Any client (producer, consumer, or admin) can issue this request to discover the current cluster state: which brokers are available, which topics exist, how partitions are distributed across brokers, which broker leads each partition, and which replicas are in sync. This information is essential for operational monitoring (detecting under-replicated partitions), capacity planning (partition distribution), debugging (leadership skew), and programmatic cluster inspection. The metadata can be fetched for all topics or filtered to a specific topic. Additionally, watermark offsets (low and high) can be fetched per partition to determine data retention boundaries.
Usage
Apply this principle when you need to inspect Kafka cluster topology at runtime: verifying topic existence before producing, monitoring partition health and replication status, detecting leadership imbalance, or building administrative dashboards. Metadata fetching is a synchronous operation with a configurable timeout.
Theoretical Basis
The metadata inspection model follows a hierarchical query pattern:
Metadata Hierarchy:
Metadata Response
├── Origin broker (id, name)
├── Brokers[]
│ ├── id: i32
│ ├── host: String
│ └── port: i32
└── Topics[]
├── name: String
├── error: Option<ErrorCode>
└── Partitions[]
├── id: i32
├── leader: i32 (broker id)
├── replicas: [i32] (broker ids)
├── isr: [i32] (in-sync broker ids)
└── error: Option<ErrorCode>
Key Diagnostic Patterns:
- Under-replicated: isr.len() < replicas.len() indicates follower lag
- Offline partition: leader == -1 indicates no available leader
- Leadership skew: Uneven distribution of leader counts across brokers
- Watermark inspection: (low_watermark, high_watermark) per partition for retention analysis