Workflow:Apache Kafka Broker Startup
| Knowledge Sources | |
|---|---|
| Domains | Operations, Infrastructure |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
End-to-end process for starting a Kafka broker from shell script invocation through JVM launch with proper classpath, logging, and runtime configuration.
Description
This workflow documents the chain of operations that occur when a Kafka broker is started using the standard bin/kafka-server-start.sh script. The process flows through the core JVM launcher (kafka-run-class.sh) which handles classpath construction, JVM option configuration, logging setup, and daemon mode handling before ultimately launching the kafka.Kafka main class. Understanding this chain is essential for troubleshooting startup issues, configuring JVM parameters, and customizing deployment environments.
Usage
Execute this workflow when you need to start a Kafka broker in KRaft mode or when troubleshooting broker startup failures. You need a compiled Kafka installation (via ./gradlew jar or a binary distribution), a server.properties configuration file, and a Java runtime (JDK 17+).
Execution Steps
Step 1: Invoke Server Start Script
Execute bin/kafka-server-start.sh with a server.properties configuration file. The script accepts an optional -daemon flag for background execution and --override flags for property overrides. It sets default Log4j2 configuration and JVM heap options if not already defined in the environment.
Key considerations:
- Default heap is -Xmx1G -Xms1G if KAFKA_HEAP_OPTS is not set
- Log4j2 configuration defaults to config/log4j2.yaml
- The -daemon flag prepends daemon mode arguments for background execution
Step 2: Construct JVM Classpath
The kafka-run-class.sh script assembles a comprehensive classpath from all Kafka module build outputs, dependency jars, and configuration directories. It scans core, clients, streams, connect, tools, and other module directories, filtering out test jars, source jars, and documentation jars unless INCLUDE_TEST_JARS is set.
Key considerations:
- Classpath assembly is Scala-version-aware (defaults to 2.13)
- Test and source jars are excluded by default via regex filtering
- The script supports both development (build/) and packaged (libs/) directory layouts
- Windows (Cygwin/MinGW) path conversion is handled automatically
Step 3: Configure JVM Options
Set up JVM performance and diagnostic options including GC logging, memory settings, and platform-specific tuning. The script configures GC log rotation, sets JAVA_OPTS for locale and headless mode, and applies any user-defined KAFKA_OPTS or KAFKA_JVM_PERFORMANCE_OPTS.
Key considerations:
- GC logging is configured with rotation (10 files, 100MB each) when -loggc is specified
- The script detects Java version to apply appropriate GC log flags
- KAFKA_JMX_OPTS can be set for JMX remote monitoring
- The LOG_DIR environment variable controls log file placement
Step 4: Launch Kafka Main Class
Execute the kafka.Kafka main class with the assembled classpath, JVM options, and the server.properties configuration file. In daemon mode, the process is backgrounded with stdout/stderr redirected to the configured log directory. In foreground mode, the process runs directly via exec.
Key considerations:
- Daemon mode uses nohup and redirects output to $LOG_DIR/kafkaServer.out
- The service name (kafkaServer) is used for process identification
- The main class initializes KRaft consensus, network listeners, and log management
- Configuration properties can be overridden via --override on the command line