Principle:Apache Kafka Broker Logging Configuration
| Knowledge Sources | |
|---|---|
| Domains | Operations, Logging, Observability |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
Principle for structuring Kafka broker log output into dedicated files per subsystem using Log4j2 appender hierarchies and logger routing.
Description
Broker Logging Configuration is the principle of separating broker log output into multiple dedicated files based on subsystem responsibility. Rather than writing all log messages to a single file, the broker routes logs from different components (requests, controller, state changes, log cleaning, authorization) to separate rolling files. This separation is critical for operational monitoring and troubleshooting in production Kafka clusters, as it allows operators to examine specific subsystem behavior without sifting through unrelated log noise.
The configuration uses Log4j2's additivity property (set to false) to ensure that logs routed to subsystem-specific appenders do not also appear in the root logger's output, preventing duplication. All rolling file appenders use hourly time-based rotation for manageable file sizes.
Usage
Use this principle when configuring logging for Kafka broker deployments. The subsystem-separated log structure should be the default for any production deployment, and operators should customize log levels per subsystem based on their monitoring and debugging needs.
Theoretical Basis
The logging architecture follows a hierarchical logger routing pattern:
Pseudo-code Logic:
# Abstract algorithm description (NOT real implementation)
# Define appenders (one per subsystem)
appenders = {
"KafkaAppender": "server.log", # General broker logs
"StateChangeAppender": "state-change.log", # Partition state transitions
"RequestAppender": "kafka-request.log", # Request handling
"CleanerAppender": "log-cleaner.log", # Log compaction
"ControllerAppender": "controller.log", # Controller operations
"AuthorizerAppender": "kafka-authorizer.log", # Authorization events
}
# Route loggers to specific appenders with additivity=false
routing = {
"kafka.request.logger": ("RequestAppender", WARN),
"org.apache.kafka.controller": ("ControllerAppender", INFO),
"...LogCleaner": ("CleanerAppender", INFO),
"state.change.logger": ("StateChangeAppender", INFO),
"kafka.authorizer.logger": ("AuthorizerAppender", INFO),
}
# Root logger catches everything else -> STDOUT + server.log
The additivity=false setting on child loggers prevents log events from propagating to parent loggers, ensuring each subsystem log file contains only its designated output.