Principle:Heibaiying BigData Notes Kafka Consumer Configuration
| Knowledge Sources | |
|---|---|
| Domains | Messaging, Distributed_Systems |
| Last Updated | 2026-02-10 10:00 GMT |
Overview
Configuring and instantiating a Kafka consumer requires setting properties that control deserialization, group membership, and offset management, followed by subscribing to one or more topics for automatic partition assignment.
Description
A Kafka consumer reads records from topic partitions. Before consuming, the consumer must be configured with a Properties object containing essential settings:
- bootstrap.servers: Broker addresses for initial cluster connection and metadata discovery.
- group.id: The consumer group identifier. All consumers sharing the same group.id coordinate to divide partitions among themselves. Each partition is assigned to exactly one consumer within a group at any time.
- key.deserializer: The class used to convert raw bytes back into the record key type. Must match the serializer used by the producer.
- value.deserializer: The class used to convert raw bytes back into the record value type.
- enable.auto.commit: When set to true (the default), the consumer automatically commits offsets at regular intervals defined by auto.commit.interval.ms. When set to false, the application must manually commit offsets.
After instantiation, the consumer must subscribe to one or more topics. Subscription triggers the group coordination protocol: the consumer joins (or creates) the consumer group, and the group coordinator assigns partitions to each consumer in the group.
Usage
Use this principle whenever you need to read messages from Kafka topics in a Java application. Consumer configuration is the mandatory first step in any consumption workflow. The choice of group.id determines whether consumers cooperate (same group, partitions divided) or independently consume all data (different groups, each sees all messages).
Theoretical Basis
The consumer group protocol works as follows:
1. Create a Properties object with required configuration.
2. Set "bootstrap.servers", "group.id", "key.deserializer", "value.deserializer".
3. Set "enable.auto.commit" to true or false based on offset management strategy.
4. Instantiate KafkaConsumer with the Properties.
5. Call consumer.subscribe(topicList) to join the consumer group.
6. The consumer sends a JoinGroup request to the group coordinator broker.
7. The coordinator assigns partitions across all consumers in the group.
8. Each consumer receives its partition assignment and begins fetching.
The partition assignment strategy is controlled by partition.assignment.strategy and defaults to RangeAssignor. Alternative strategies include RoundRobinAssignor and StickyAssignor.
Key differences between subscription and manual assignment:
- subscribe(): Enables automatic rebalancing when consumers join or leave the group. Partitions are dynamically reassigned.
- assign(): Manually assigns specific partitions. No group coordination or rebalancing occurs. Useful for stateful processing that requires stable partition ownership.