What happens after a Kafka producer calls send()?

This sounds like a simple question. The producer creates a record, sends it to Kafka, and receives an acknowledgement.

However, quite a lot can happen between these steps.

The producer may wait for more records and create a batch. It must choose a partition. It may compress the batch. If the broker is slow, the producer may run out of buffer space. If a request fails, it may retry it. If the retry succeeds after the first request was already written, Kafka may receive the same batch twice.

Kafka Simulator v1.1 focuses on these producer semantics.

The release adds 10 new scenarios, bringing the complete curriculum to 23 of 125 scenarios.

What is included in v1.1?

Eight of the ten scenarios focus on the producer. The remaining two extend the log-anatomy track: reading up to the high watermark, and the log start offset.

The producer scenarios cover five areas:

  • batching with linger.ms and batch.size,
  • producer partitioners,
  • compression,
  • backpressure with buffer.memory and max.block.ms,
  • retries and idempotence.

Let us look at each of them.

Batching with linger.ms and batch.size

A Kafka producer does not necessarily send every record in a separate request.

Records assigned to the same partition can be collected into a batch. The producer sends the batch when it becomes full or when the configured waiting time expires.

This is where two producer settings become important:

  • batch.size limits the size of a batch,
  • linger.ms allows the producer to wait for additional records before sending it.

The trade-off is between latency and throughput.

A larger batch usually means fewer requests and better compression. However, waiting for the batch to fill may increase the latency of an individual record.

The new scenarios make this behaviour visible. You can see when records enter a batch, what causes the batch to be sent, and how changing the configuration affects the requests produced by the client.

Producer partitioners

Before the producer can create a partition-specific batch, it must select a partition.

For keyed records, the key is used to choose the partition. Records with the same key are therefore normally assigned to the same partition.

For records without a key, the behaviour is different. The producer can group records using its partitioning strategy instead of calculating the partition from a key.

The simulator shows both cases step by step:

  1. the producer receives the record,
  2. the partitioner selects a partition,
  3. the record is added to the corresponding batch,
  4. the batch is sent to that partition’s leader.

This makes it easier to see why partitioning affects both ordering and batching.

Compression happens per batch

Kafka producer compression is applied to record batches, not independently to every record.

This distinction matters.

Compression is usually more effective when a batch contains multiple similar records. A batch containing only one small record may gain very little from compression. A larger batch with repetitive data may be compressed much more efficiently.

As a result, compression is connected to batching configuration. Settings such as batch.size and linger.ms can influence not only request frequency, but also the shape and compressibility of the resulting batches.

The new scenarios show the uncompressed and compressed batch sizes, making this relationship explicit.

Backpressure with buffer.memory and max.block.ms

What happens when an application produces records faster than Kafka can accept them?

The producer cannot keep an unlimited number of unsent records in memory. It uses a bounded buffer, configured using buffer.memory.

When the buffer has no available space, a call to send() may block while waiting for memory to be released.

It will not wait forever. The maximum blocking time is controlled by max.block.ms.

This gives us two possible outcomes:

  • enough data is sent and acknowledged, freeing buffer space,
  • no space becomes available before the timeout, and the send fails.

The simulator tracks the producer buffer directly. You can see memory being allocated when records are added and released as requests complete.

This is especially useful when combined with a slow broker or a throughput quota. Instead of treating producer backpressure as an abstract configuration issue, you can watch the buffer fill up and see the exact send that becomes blocked.

Retries and the duplicate problem

Suppose the producer sends a batch to a broker.

The broker writes the batch successfully, but the acknowledgement is lost before it reaches the producer.

From the producer’s perspective, the request failed. It cannot know whether the broker wrote the data. Retrying is therefore reasonable.

However, if the first request was written successfully, the retry may append the same batch again.

This is the duplicate problem caused by ambiguous failures.

The retry scenario shows both requests:

  1. the original request reaches the broker,
  2. the acknowledgement is lost,
  3. the producer retries the batch,
  4. the broker receives the same records for the second time.

Without idempotence, both copies can be appended.

Idempotent producer semantics

An idempotent producer allows Kafka to recognise retried batches.

Kafka assigns the producer an identity and tracks sequence numbers independently for each partition. When a batch is retried with a sequence number that the broker has already processed, the broker can acknowledge it without appending the records again.

The important distinction is between a retry and a genuine duplicate.

Two records may contain exactly the same key and value and still be two separate records intentionally sent by the application. Kafka must not remove such records.

Idempotence deduplicates producer protocol retries. It does not deduplicate application data based on its contents.

The new scenarios show:

  • producer IDs,
  • partition sequence numbers,
  • successful idempotent retries,
  • genuine duplicate records,
  • out-of-order sequence detection.

Why does max.in.flight.requests.per.connection matter?

A producer can have multiple requests waiting for responses at the same time.

This improves throughput, but it also creates a potential ordering problem.

Suppose the producer sends batch A and then batch B. Batch A fails temporarily, while batch B succeeds. If A is retried after B has already been written, the records may be appended in a different order.

Idempotence uses sequence numbers to detect invalid ordering.

The simulator shows this explicitly. You can see the expected broker sequence number, the number carried by the incoming batch, and the point at which the broker rejects an out-of-order request.

Inspecting internal producer state

The new scenarios lean heavily on the simulator’s inspector, which exposes internal state for producers and partitions, including:

  • retry counters,
  • producer sequence numbers,
  • broker sequence numbers,
  • buffer allocation.

Normally, these values are hidden behind producer metrics, logs, and protocol behaviour. In the simulator, they can be inspected directly while advancing or rewinding a scenario.

Producer model fixes

Adding the new scenarios also required making the existing producer model more precise.

Retries are counted per batch

Kafka retries produce requests containing batches of records.

A failed request should therefore consume one retry for the complete batch, not one retry for every record inside it.

The simulator now follows this model. Every record from a failed batch is sent again, while the retry counter advances once.

Buffer memory is released precisely

Producer buffer memory is now released as acknowledgements arrive.

This fixes cases where the simulator could report backpressure even though acknowledged records should already have released their memory.

The same applies to max.block.ms timeouts. When a blocked send times out, its buffer allocation is cleaned up.

acks=all is tracked per partition

Kafka durability is defined independently for every partition.

A producer writing to multiple partitions must therefore wait for the correct partition to satisfy its acknowledgement requirements. Progress in another partition must not acknowledge an unrelated request.

The simulator now tracks this state separately for every partition and topic.

Producer reset clears idempotence state

The broker stores deduplication state associated with an idempotent producer.

When a simulated producer is removed or reset, this state must also be removed. Otherwise, a newly created producer may have its records incorrectly classified as old retries.

The reset behaviour now clears the corresponding broker-side state.

Rewinding restores retry state

Scenarios can be advanced and rewound one step at a time.

Rewinding now reconstructs the retry counters used during each simulation round. The visible producer state is therefore consistent regardless of whether the scenario is moving forward or backward.

Playground improvements

The release also contains several fixes outside the producer model.

Consumer group rebalancing

Consumer groups in the playground received several correctness fixes:

  • the sticky assignor now minimises partition movement instead of reassigning from scratch,
  • a static member that stays away longer than session.timeout.ms is evicted like any other member,
  • changing a topic’s partition count now triggers a proper rebalance without disturbing member state,
  • partitions are never assigned to a member that is no longer alive.

Topic configuration and compaction

Kafka never reduces the partition count of a topic. Shrinking would delete data and break key-based partitioning, so kafka-topics --alter rejects it with InvalidPartitionsException.

The playground now follows the same rule. A partition-count decrease is rejected with a teaching error instead of silently dropping partitions.

Compacted topics also behave more precisely:

  • a tombstone survives until its own delete.retention.ms grace window expires,
  • a keyless record cannot be produced to a compacted topic,
  • with compact,delete, time retention now reclaims aged records independently of compaction.

The playground serialises its actions into the URL so that a scenario can be shared with another person.

The codec now includes all configuration changes, including partition-count updates. A link that requests a cluster type from a later release pack now falls back to a supported topology instead of rendering an unusable cluster.

Opening a shared link should therefore recreate the same scenario that was originally built.

Interaction fixes

Several smaller playground issues were also corrected:

  • entity identifiers no longer collide after an entity is removed and recreated,
  • producer placement works correctly on wider broker grids.

Security headers

The site now sends a Content Security Policy and a Permissions Policy.

Why start with the producer?

Kafka guarantees are often discussed from the broker’s perspective: replication, ISR membership, high watermarks, and leader failover.

However, the guarantee observed by an application also depends on the producer.

Which acknowledgement mode does it use? Does it retry? Is idempotence enabled? How many requests can be in flight? What happens when its local buffer becomes full?

These settings determine whether a write is fast, durable, ordered, duplicated, blocked, or rejected.

This is why the first larger Kafka Simulator scenario pack focuses on the producer.

Once batching, acknowledgements, retries, and idempotence are visible, later scenarios covering replication, durability, and transactions have a clearer foundation.

Open the Kafka Simulator, select the producer track, and try the failure scenarios step by step.