Workflow:Apache Kafka Topic Management
| Knowledge Sources | |
|---|---|
| Domains | Operations, Administration |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
End-to-end process for managing Kafka topics through the CLI, covering creation, listing, describing, altering configuration, and deletion via the kafka-topics.sh tool and the AdminClient API.
Description
This workflow documents the standard operations for managing Kafka topics using the bin/kafka-topics.sh command-line tool. The tool delegates to the TopicCommand Java class which uses the KafkaAdminClient to communicate with the Kafka cluster. It supports creating topics with specified partitions and replication factors, listing existing topics, describing topic details including partition assignments and configuration, altering topic configuration, and deleting topics. The AdminClient handles broker discovery, request routing, and retry logic transparently.
Usage
Execute this workflow when you need to create, inspect, modify, or remove topics in a running Kafka cluster. You need a running Kafka cluster with accessible bootstrap servers and appropriate ACL permissions if authorization is enabled.
Execution Steps
Step 1: Bootstrap Connection
The kafka-topics.sh script invokes kafka-run-class.sh to launch the TopicCommand class. The tool parses command-line arguments to determine the operation (--create, --list, --describe, --alter, --delete) and establishes an AdminClient connection to the cluster using the --bootstrap-server parameter.
Key considerations:
- The --bootstrap-server flag is required for all operations
- The AdminClient discovers cluster topology via metadata requests
- Additional client configuration can be passed via --command-config for security settings
Step 2: Create Topic
Create a new topic with specified partitions, replication factor, and optional configuration overrides. The AdminClient sends a CreateTopics request to the cluster controller, which validates the request against any configured CreateTopicPolicy and assigns partitions to brokers.
Key considerations:
- --partitions and --replication-factor control topic layout
- --config allows setting topic-level configuration (e.g., retention, compression)
- Rack-aware replica assignment is applied automatically when broker rack IDs are configured
- The --if-not-exists flag prevents errors when the topic already exists
Step 3: List Topics
Retrieve and display all topic names in the cluster. The AdminClient sends a Metadata request and filters the response to show non-internal topics by default.
Key considerations:
- Internal topics (prefixed with __) are hidden by default
- The --include flag supports regex patterns for filtering topic names
- The --exclude-internal flag explicitly filters system topics
Step 4: Describe Topic
Display detailed information about one or more topics including partition count, replication factor, per-partition leader and replica assignments, ISR (in-sync replica) lists, and any configuration overrides. The AdminClient combines DescribeTopics and DescribeConfigs requests.
Key considerations:
- Use --topic to describe specific topics or omit for all topics
- The output shows partition leader, replicas, and ISR for each partition
- --under-replicated-partitions and --unavailable-partitions filter problem partitions
- Topic configuration overrides are displayed alongside defaults
Step 5: Alter Topic Configuration
Modify topic-level configuration properties using the kafka-configs.sh tool (not kafka-topics.sh directly for config changes). The AdminClient sends AlterConfigs or IncrementalAlterConfigs requests to update topic settings dynamically without restart.
Key considerations:
- Configuration changes take effect immediately without broker restart
- Use kafka-configs.sh with --entity-type topics for configuration management
- IncrementalAlterConfigs (default in newer versions) allows modifying individual properties
Step 6: Delete Topic
Remove a topic and all its data from the cluster. The AdminClient sends a DeleteTopics request to the controller, which marks the topic for deletion and asynchronously removes partition data from all brokers.
Key considerations:
- Topic deletion is asynchronous; data removal happens in the background
- The --if-exists flag prevents errors when the topic does not exist
- Delete operations require appropriate ACL permissions when authorization is enabled
- Internal topics cannot be deleted through this interface