Environment:Fede1024 Rust rdkafka Kafka Broker Runtime

Knowledge Sources	rust-rdkafka Apache Kafka
Domains	Infrastructure, Messaging
Last Updated	2026-02-07 19:30 GMT

Overview

Apache Kafka broker (0.8.x through 4.0) accessible on `localhost:9092`, provisioned via Docker Compose in KRaft mode for integration testing and development.

Description

This environment provides a running Apache Kafka broker required by any code that actually connects to Kafka (producers, consumers, admin clients). For development and testing, the project provides a `docker-compose.yaml` that starts a single-node Kafka broker using the Bitnami image in KRaft mode (no ZooKeeper dependency). The broker listens on port 9092 with PLAINTEXT protocol. Production deployments require a properly configured Kafka cluster.

Usage

Use this environment for integration testing, running examples, or development against a live Kafka broker. It is required by the test suite (`test_suite.sh`) and all example programs. The MockCluster implementations do NOT require this environment as they simulate Kafka in-process.

System Requirements

Category	Requirement	Notes
Runtime	Docker Engine + Docker Compose	Required for local development/testing broker
Network	Port 9092 (PLAINTEXT), 9093 (CONTROLLER)	Must be available on localhost
Kafka Version	0.8.x through 4.0	CI tests against 3.7, 3.8, 3.9, 4.0
Disk	Moderate	Kafka data and Docker image storage

Dependencies

System Packages

`docker` - Container runtime for Kafka broker
`docker-compose` (or `docker compose` v2) - Orchestration for Kafka service

Docker Image

`bitnamilegacy/kafka:${KAFKA_VERSION:-4.0}` - Default Kafka 4.0 in KRaft mode

Kafka Broker Configuration (docker-compose.yaml)

`KAFKA_CFG_PROCESS_ROLES=broker,controller` - Combined mode (single node)
`KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093`
`KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092`
`KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR=1` - Single broker
`KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1` - Single broker
`KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR=1` - Single broker
`KAFKA_CFG_NUM_PARTITIONS=3` - Default partition count
`KAFKA_KRAFT_ENABLED=true` - No ZooKeeper required

Credentials

The following environment variables control the runtime:

`KAFKA_VERSION`: Kafka version to use (default: `4.0`). Valid values: `3.7`, `3.8`, `3.9`, `4.0`.
`RUST_LOG`: Logging configuration for tests (e.g., `librdkafka=trace,rdkafka::client=debug`).
`RUST_BACKTRACE`: Set to `1` to enable backtraces in test output.

No authentication credentials are required for the default PLAINTEXT configuration.

Quick Install

# Start Kafka broker (default version 4.0)
docker compose up -d

# Or specify a version
KAFKA_VERSION=3.9 docker compose up -d

# Run integration tests
./test_suite.sh

# Teardown
docker compose down

Code Evidence

Docker Compose configuration from `docker-compose.yaml:1-23`:

services:
  kafka:
    image: bitnamilegacy/kafka:${KAFKA_VERSION:-4.0}
    environment:
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
      - KAFKA_CFG_NUM_PARTITIONS=3
      - KAFKA_KRAFT_ENABLED=true
    ports: ["9092:9092"]

CI test matrix from `.github/workflows/ci.yml:71-95`:

test:
    strategy:
      fail-fast: false
      max-parallel: 1  # Test suite doesn't support concurrent runs
      matrix:
        include:
          - kafka-version: "4.0"
          - kafka-version: "3.9"
          - kafka-version: "3.8"
          - kafka-version: "3.7"

Test suite environment setup from `test_suite.sh:31-36`:

git submodule update --init
docker compose up -d
# ...
RUST_BACKTRACE=1 cargo test "${@}"

Common Errors

Error Message	Cause	Solution
Connection refused on port 9092	Kafka broker not running	Run `docker compose up -d` and wait for broker to be ready
`BrokerTransportFailure`	Network issue or broker crashed	Check `docker compose logs kafka` for errors
`GroupCoordinatorNotAvailable`	Broker still initializing	Wait a few seconds after container start for broker to become ready
Test failures with concurrent runs	Test suite not designed for parallel execution	CI uses `max-parallel: 1`; run tests sequentially

Compatibility Notes

KRaft Mode: The docker-compose configuration uses KRaft (no ZooKeeper), which requires Kafka 3.3+. Older Kafka versions would need a ZooKeeper-based configuration.
Single Broker: The development setup uses a single broker with replication factor 1. Production deployments need multiple brokers.
PLAINTEXT Only: No SSL/TLS or SASL authentication configured in the development setup. Production deployments should use appropriate security protocols.
Partition Count: Default is 3 partitions. Some tests may depend on this specific value.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment