Scenario 04.3.3 · exactly_once_v2 end-to-end

Kafka processing.guarantee=exactly_once_v2 end to end

Transactions · Advanced · about 4 min · Apache Kafka 4.3

Full EOS on a pipeline: idempotent output, output + input offsets committed atomically, and a stale-epoch zombie producer fenced - the three guarantees of processing.guarantee=exactly_once_v2.

What you start with

The full exactly_once_v2 stack on a consume-process-produce pipeline, combining its three guarantees: idempotent output (dedup on retry), transactional atomicity of the output records together with the input offsets, and zombie fencing. Under exactly_once_v2 Kafka Streams fences zombies mainly through the consumer group generation sent with each transactional offset commit (KIP-447); the simulator shows the transactional.id epoch check instead, with a stale-epoch zombie attempting to write after the pipeline commits.

Step by step

1. Cluster ready · EOS v2 t+0.000s The full exactly_once_v2 stack: an idempotent transactional producer `tx` (epoch 1), its stale zombie `tx-zombie` (epoch 0, same transactional.id), a read_committed pipeline consumer, and input/output topics. Open at this step: step 1, Cluster ready · EOS v2
2. Input seeded · orders-in t+0.600s Two input records are seeded on `orders-in` (immediately stable), ready for the pipeline to consume. Open at this step: step 2, Input seeded · orders-in
3. Polled · processed t+1.500s The pipeline group polled and advanced to 2 - both inputs processed. Idempotence will guard the write side; the transaction guards the offset side. Open at this step: step 3, Polled · processed
4. begin output txn t+1.650s `tx` - the live, highest-epoch instance - begins a transaction. The coordinator epoch for transactional.id `tx-eos` is 1. Open at this step: step 4, begin output txn
5. Output produced · uncommitted t+2.300s The output record lands on `orders-out` (HW = 1) but the LSO stays at 0, held until commit. Because the producer is idempotent, a retried produce here would be deduplicated by the broker, never doubled. Open at this step: step 5, Output produced · uncommitted
6. Offsets staged t+2.500s send_offsets_to_txn binds the input position {0:2} to the transaction, so the output and the offsets will commit together. Open at this step: step 6, Offsets staged
7. commit · output + offsets t+3.400s Atomic commit - the output LSO advances to 1 and the input offsets commit to {0:2}. Idempotent writes, output, and offsets all land exactly once. Open at this step: step 7, commit · output + offsets
8. zombie fenced t+4.000s The old-epoch `tx-zombie` wakes and tries begin_transaction. The coordinator holds epoch 1, so the epoch-0 instance is FENCED (producer_fenced, status error) and can never corrupt the committed stream. In Kafka the fence lands on its first send or commit, since beginTransaction() is client-side, and transaction.version=2 bumps the epoch at every commit. Open at this step: step 8, zombie fenced

The takeaway

Together - idempotence, atomic offset-with-output commit, and epoch fencing - these give processing.guarantee=exactly_once_v2: no duplicate output, no double-processing or loss, and no zombie corrupting the stream.

Configs in this scenario

• transactional.id A stable name that turns on transactions for a producer (and implies idempotence); a new producer that calls initTransactions() with the same id fences the older one.

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=04.3.3" title="Kafka Simulator: exactly_once_v2 end-to-end" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/exactly-once-v2-end-to-end/">Kafka processing.guarantee=exactly_once_v2 end to end</a></p>

Further reading

Apache Kafka documentation

Troubleshooting

• ProducerFencedException in Kafka: causes and fixes

More in Transactions

• Kafka at-most-once vs at-least-once vs exactly-once
• At-least-once delivery in Kafka: where duplicates come from
• Kafka transaction commit vs abort with read_committed
• Kafka isolation.level: read_committed vs read_uncommitted
• Kafka last stable offset (LSO) with concurrent transactions
• Kafka transaction.timeout.ms: the coordinator aborts
• Kafka read_committed during a leader election
• Kafka consume-process-produce loop with exactly-once
• Kafka zombie producer fencing with transactional.id

All scenarios