Principle:Heibaiying BigData Notes Kafka Rebalancing and Shutdown
| Knowledge Sources | |
|---|---|
| Domains | Messaging, Distributed_Systems |
| Last Updated | 2026-02-10 10:00 GMT |
Overview
Handling partition rebalancing and graceful consumer shutdown ensures that offsets are committed before partitions are revoked and that the consumer exits cleanly without triggering unnecessary rebalances.
Description
Rebalancing occurs when the set of consumers in a group changes (a consumer joins, leaves, or is considered dead) or when topic metadata changes (partitions are added). During a rebalance, partitions are reassigned across the active consumers. This has two critical implications:
- Partition revocation: A consumer may lose ownership of partitions it was previously processing. Any uncommitted offsets for those partitions will be lost, leading to duplicate processing when the partition is reassigned to another consumer.
- Partition assignment: A consumer may receive new partitions that were previously owned by another consumer. It must begin consuming from the last committed offset for those partitions.
The ConsumerRebalanceListener interface provides two callback methods:
- onPartitionsRevoked(Collection<TopicPartition> partitions): Called before partitions are taken away from this consumer. This is the opportunity to commit offsets for the partitions being revoked.
- onPartitionsAssigned(Collection<TopicPartition> partitions): Called after new partitions are assigned to this consumer. This is the opportunity to initialize state or seek to specific offsets for the newly assigned partitions.
Graceful shutdown is achieved using the consumer.wakeup() method, which is the only KafkaConsumer method that is safe to call from another thread. When wakeup() is called, the next call to poll() (or a currently blocking poll()) throws a WakeupException. The consumer catches this exception, commits offsets, and closes cleanly.
Usage
Use ConsumerRebalanceListener whenever your application performs manual offset commits and needs to guarantee no data loss during rebalances. Use consumer.wakeup() in combination with a shutdown hook or signal handler to implement graceful shutdown that properly commits offsets and releases consumer group membership.
Theoretical Basis
The rebalance and shutdown patterns work together:
Rebalance Handling:
1. Consumer subscribes with a ConsumerRebalanceListener.
2. When a rebalance is triggered:
a. onPartitionsRevoked() is called with the partitions being taken away.
b. The listener commits offsets for the revoked partitions (commitSync).
c. Partitions are reassigned by the group coordinator.
d. onPartitionsAssigned() is called with the newly assigned partitions.
e. The consumer resumes polling with its new assignment.
Graceful Shutdown:
1. A shutdown hook or signal handler calls consumer.wakeup().
2. The current or next poll() throws WakeupException.
3. The consumer catches WakeupException in the poll loop.
4. In the finally block:
a. consumer.commitSync() commits final offsets.
b. consumer.close() sends a LeaveGroup request to the coordinator.
5. The coordinator triggers an immediate rebalance for the remaining consumers.
Without wakeup(), a consumer that simply stops calling poll() will not be detected as dead until session.timeout.ms expires, delaying the rebalance. Using wakeup() followed by close() triggers an immediate, clean departure from the group.
The WakeupException is an unchecked exception that should not be caught inside the poll loop body. Instead, it should propagate to the outer try-catch block that handles shutdown.