Implementation:DataTalksClub Data engineering zoomcamp Kafka Docker Compose Setup

Page Metadata
Knowledge Sources	repo: DataTalksClub/data-engineering-zoomcamp
Domains	Data_Engineering, Stream_Processing
Last Updated	2026-02-09 14:00 GMT

Overview

Concrete tool documentation for provisioning a Kafka streaming infrastructure stack using Docker Compose with Confluent Platform images, including broker, ZooKeeper, Schema Registry, Control Center, and REST Proxy services on an external network.

Description

This Docker Compose configuration defines a five-service Kafka infrastructure stack built on Confluent Platform 7.2.0 images. The stack provisions:

broker (confluentinc/cp-kafka:7.2.0): The Apache Kafka message broker, exposed on port 9092 for host access and port 29092 for inter-container communication. Configured with dual listeners (PLAINTEXT for internal, PLAINTEXT_HOST for external) and single-node replication settings.
zookeeper (confluentinc/cp-zookeeper:7.2.0): The coordination service for broker metadata and leader election, exposed on port 2181.
schema-registry (confluentinc/cp-schema-registry:7.2.0): The centralized schema repository, exposed on port 8081. Connects to the broker via the internal listener at broker:29092.
control-center (confluentinc/cp-enterprise-control-center:7.2.0): The Confluent Control Center web UI for monitoring and managing Kafka clusters, exposed on port 9021.
kafka-rest (confluentinc/cp-kafka-rest:7.2.0): The Kafka REST Proxy for HTTP-based produce and consume operations, exposed on port 8082.

All services attach to an external Docker network named kafka-spark-network, allowing other independently managed stacks (such as a Spark cluster) to communicate with the broker.

Usage

Use this implementation to:

Stand up a local Kafka development and testing environment with a single command.
Enable schema-governed message production and consumption via the Schema Registry.
Monitor topic throughput, consumer lag, and broker health through the Control Center web UI.
Connect Spark Structured Streaming jobs to the broker over the shared kafka-spark-network.

Code Reference

Source Location

File	`07-streaming/python/docker/kafka/docker-compose.yml`
Lines	L1-82
Repository	DataTalksClub/data-engineering-zoomcamp

Signature

docker-compose -f 07-streaming/python/docker/kafka/docker-compose.yml up -d

Import

No application-level imports. Requires Docker Engine and Docker Compose installed on the host.

I/O Contract

Inputs

Name	Type	Description
docker-compose.yml	YAML file	Declarative service definitions for the Kafka stack
kafka-spark-network	Docker network (external)	Pre-existing external network; must be created before running: `docker network create kafka-spark-network`

Outputs

Service	Port	Protocol	Description
broker	9092 (host), 29092 (internal)	Kafka binary protocol	Message broker accepting produce/consume connections
zookeeper	2181	ZooKeeper protocol	Coordination service for broker metadata
schema-registry	8081	HTTP	Schema management REST API
control-center	9021	HTTP	Web UI for cluster monitoring
kafka-rest	8082	HTTP	REST Proxy for HTTP-based produce/consume

Usage Examples

Step 1: Create the external network (one-time setup)

docker network create kafka-spark-network

Step 2: Start the Kafka stack

cd 07-streaming/python/docker/kafka
docker-compose up -d

Step 3: Verify all services are running

docker-compose ps

Expected output shows all five services in the "Up" state:

     Name                   Command               State                Ports
-----------------------------------------------------------------------------------------
broker             /etc/confluent/docker/run   Up      0.0.0.0:9092->9092/tcp
control-center     /etc/confluent/docker/run   Up      0.0.0.0:9021->9021/tcp
kafka-rest         /etc/confluent/docker/run   Up      0.0.0.0:8082->8082/tcp
schema-registry    /etc/confluent/docker/run   Up      0.0.0.0:8081->8081/tcp
zookeeper          /etc/confluent/docker/run   Up      0.0.0.0:2181->2181/tcp

Step 4: Create a test topic via the broker container

docker exec broker kafka-topics \
    --bootstrap-server broker:29092 \
    --create \
    --topic test-topic \
    --partitions 2 \
    --replication-factor 1

Step 5: Access the Control Center web UI

Open a browser and navigate to http://localhost:9021 to view the cluster dashboard.

Step 6: Shut down the stack

docker-compose down

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment