Principle:Heibaiying BigData Notes Kafka Message Sending
| Knowledge Sources | |
|---|---|
| Domains | Messaging, Distributed_Systems |
| Last Updated | 2026-02-10 10:00 GMT |
Overview
Kafka supports three distinct patterns for sending messages: fire-and-forget, synchronous (blocking), and asynchronous (callback-based), each offering different trade-offs between simplicity, reliability, and throughput.
Description
Once a KafkaProducer is instantiated, messages are sent by constructing a ProducerRecord and invoking the send() method. A ProducerRecord encapsulates the target topic, an optional partition, an optional key, and the message value. The producer internally handles serialization, partitioning, batching, and network transmission.
The three sending patterns are:
- Fire-and-forget: The send() method is called without inspecting the returned Future or providing a callback. This offers the highest throughput but provides no confirmation that the message was successfully delivered. Suitable for metrics or logging where occasional message loss is acceptable.
- Synchronous send: The send() method returns a Future<RecordMetadata>. Calling .get() on this future blocks the calling thread until the broker acknowledges the record (or an error occurs). This guarantees that each message is confirmed before proceeding, at the cost of reduced throughput due to blocking.
- Asynchronous send: A Callback is passed as the second argument to send(). The callback is invoked when the broker acknowledges the record (or when an error occurs), without blocking the calling thread. This combines high throughput with reliable delivery confirmation.
Usage
Use fire-and-forget when message loss is tolerable and maximum throughput is required. Use synchronous send when strict ordering or guaranteed delivery per message is essential (e.g., financial transactions). Use asynchronous send as the general-purpose pattern that balances throughput with delivery confirmation.
Theoretical Basis
The sending workflow follows this sequence:
1. Construct a ProducerRecord(topic, key, value).
2. The producer serializes the key and value using configured serializers.
3. The partitioner determines the target partition (based on key hash or round-robin).
4. The record is added to an internal batch buffer for the target partition.
5. A background I/O thread sends batches to the partition leader broker.
6. The broker writes the record to the partition log and sends an acknowledgment.
7. Depending on the send pattern:
a. Fire-and-forget: No acknowledgment is checked.
b. Synchronous: Future.get() blocks until acknowledgment or error.
c. Asynchronous: Callback.onCompletion() is invoked with metadata or exception.
The RecordMetadata returned on success contains the topic, partition, offset, and timestamp of the written record. On failure, the exception indicates whether the error is retriable (e.g., NotLeaderForPartitionException) or fatal (e.g., SerializationException).
Synchronous sending effectively reduces the batch size to one, since each send blocks until the previous completes. Asynchronous sending allows the producer to continue batching while acknowledgments arrive in the background.