Environment:Heibaiying BigData Notes Kafka 2 2 Environment

Knowledge Sources	BigData-Notes Apache Kafka
Domains	Infrastructure, Messaging
Last Updated	2026-02-10 10:00 GMT

Overview

Apache Kafka 2.2.0 messaging environment with Scala 2.12, providing distributed publish-subscribe messaging for streaming pipelines.

Description

This environment provides Apache Kafka 2.2.0 compiled against Scala 2.12. It includes the Kafka broker, producer, and consumer Java client libraries (`kafka-clients:2.2.0`). Kafka runs on top of Apache Zookeeper for cluster coordination. The repository examples demonstrate both producer (sync/async) and consumer (group/standalone) patterns.

Usage

Use this environment for any Kafka Producer/Consumer operations. It is the mandatory prerequisite for the Kafka Producer Consumer Pipeline workflow, Storm-Kafka integration, and Flink-Kafka integration modules.

System Requirements

Category	Requirement	Notes
OS	Linux (CentOS 7.6 recommended)	Any Linux with JDK 8
Java	JDK 1.8+	Required by Kafka 2.2.0
Zookeeper	3.4.13+	Required for Kafka broker coordination
Hardware	Minimum 2GB RAM	Per broker node
Disk	10GB+	For Kafka log storage

Dependencies

System Packages

`kafka` = 2.2.0 (Scala 2.12 build: `kafka_2.12-2.2.0`)
`zookeeper` >= 3.4.13
`java-1.8.0-openjdk-devel`

Java Packages (Maven)

`org.apache.kafka:kafka-clients` = 2.2.0
`org.apache.kafka:kafka_2.12` = 2.2.0
`org.slf4j:slf4j-nop` = 1.7.25

Credentials

No API credentials required for basic Kafka usage. For secured clusters:

`KAFKA_OPTS`: JVM options for JAAS configuration (if SASL enabled)

Quick Install

# Download Kafka 2.2.0 (Scala 2.12)
wget https://archive.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz
tar -xzf kafka_2.12-2.2.0.tgz -C /opt/

# Start Zookeeper (if not already running)
/opt/kafka_2.12-2.2.0/bin/zookeeper-server-start.sh config/zookeeper.properties

# Start Kafka broker
/opt/kafka_2.12-2.2.0/bin/kafka-server-start.sh config/server.properties

# Maven dependency:
# <dependency>
#   <groupId>org.apache.kafka</groupId>
#   <artifactId>kafka-clients</artifactId>
#   <version>2.2.0</version>
# </dependency>

Code Evidence

Kafka client dependency from `kafka-basis/pom.xml`:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.12</artifactId>
    <version>2.2.0</version>
</dependency>

Producer bootstrap config from `SimpleProducer.java`:

props.put("bootstrap.servers", "hadoop001:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Common Errors

Error Message	Cause	Solution
`Connection to node -1 could not be established`	Kafka broker not reachable	Verify broker is running and `bootstrap.servers` points to correct host:port
`NotEnoughReplicasException`	ISR count below `min.insync.replicas`	Ensure enough brokers are running; adjust `min.insync.replicas`
`Offset commit failed`	Consumer group rebalancing	Use async commit with callback for retry logic

Compatibility Notes

Naming convention: `kafka_2.12-2.2.0.tgz` means Scala 2.12, Kafka version 2.2.0. The first number is the Scala version, not the Kafka version.
Storm-Kafka integration uses `storm-kafka-client:1.2.2` with `kafka-clients:2.2.0`.
Flink-Kafka integration uses `flink-connector-kafka_2.11:1.9.0` which is compatible with Kafka 2.2.0.
Provide at least 2 broker addresses in `bootstrap.servers` for fault tolerance (producer auto-discovers all brokers from given list).

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment