Scenario 04.3.2 · EOS - offsets in a txn + fencing
Kafka zombie producer fencing with transactional.id
Consume-process-produce commits input offsets inside the output transaction; a stale-epoch zombie producer is fenced.
Run this scenario in the playgroundCluster ready · EOS app. t+0.000s
orders has 1 partition, RF=3. Group eos is read_committed with enable.auto.commit=off. tx is the live producer instance (epoch 1); tx-zombie is an old instance of the same transactional.id (epoch 0). Press ▶.
Open at this step: step 1, Cluster ready · EOS appbegin txn 1. t+0.100s
tx opens transaction 1 (transactional.id tx-app).
Open at this step: step 2, begin txn 1Produced · uncommitted. t+1.100s
tx wrote two records; the HW advanced but the LSO is frozen, so read_committed eos sees nothing yet.
Open at this step: step 3, Produced · uncommittedtxn 1 committed · read, no auto-commit. t+1.900s
commit advances the LSO and eos reads both records (position 2). But committed stays at 0: enable.auto.commit is off, so offsets move only through the transaction.
Open at this step: step 4, txn 1 committed · read, no auto-committxn 2 · offsets staged. t+2.900s
In a second txn tx produced an output record and called sendOffsetsToTransaction(eos): eos offsets {0:2} are staged inside the transaction, not yet committed.
Open at this step: step 5, txn 2 · offsets stagedcommit · offsets + output atomic. t+3.400s
commit_transaction commits the output record AND eos input offsets atomically - eos.committed jumps to {0:2}. On an abort neither would commit: exactly-once.
Open at this step: step 6, commit · offsets + output atomiczombie fenced. t+3.900s
tx-zombie (epoch 0) tries to open a new transaction, but tx holds a newer epoch, so the zombie is FENCED and can never write duplicates. In Kafka the fence lands on its first send or commit, since beginTransaction() is client-side, and transaction.version=2 bumps the epoch at every commit.
Open at this step: step 7, zombie fencedWhat you start with
Exactly-once semantics with producer fencing. A transactional producer runs a consume-process-produce transaction - committing the output record and the input offsets together - then a stale-epoch 'zombie' instance sharing the same transactional.id wakes up and tries to begin its own transaction.
Step by step
- Cluster ready · EOS app t+0.000s orders has 1 partition, RF=3. Group eos is read_committed with enable.auto.commit=off. tx is the live producer instance (epoch 1); tx-zombie is an old instance of the same transactional.id (epoch 0). Press ▶. Open at this step: step 1, Cluster ready · EOS app
- begin txn 1 t+0.100s tx opens transaction 1 (transactional.id tx-app). Open at this step: step 2, begin txn 1
- Produced · uncommitted t+1.100s tx wrote two records; the HW advanced but the LSO is frozen, so read_committed eos sees nothing yet. Open at this step: step 3, Produced · uncommitted
- txn 1 committed · read, no auto-commit t+1.900s commit advances the LSO and eos reads both records (position 2). But committed stays at 0: enable.auto.commit is off, so offsets move only through the transaction. Open at this step: step 4, txn 1 committed · read, no auto-commit
- txn 2 · offsets staged t+2.900s In a second txn tx produced an output record and called sendOffsetsToTransaction(eos): eos offsets {0:2} are staged inside the transaction, not yet committed. Open at this step: step 5, txn 2 · offsets staged
- commit · offsets + output atomic t+3.400s commit_transaction commits the output record AND eos input offsets atomically - eos.committed jumps to {0:2}. On an abort neither would commit: exactly-once. Open at this step: step 6, commit · offsets + output atomic
- zombie fenced t+3.900s tx-zombie (epoch 0) tries to open a new transaction, but tx holds a newer epoch, so the zombie is FENCED and can never write duplicates. In Kafka the fence lands on its first send or commit, since beginTransaction() is client-side, and transaction.version=2 bumps the epoch at every commit. Open at this step: step 7, zombie fenced
The takeaway
The coordinator fences the zombie because only the latest producer epoch may write - a partitioned-then-recovered task can never corrupt the committed stream.
Configs in this scenario
-
transactional.A stable name that turns on transactions for a producer (and implies idempotence); a new producer that calls initTransactions() with the same id fences the older one.id -
enable.Whether the consumer commits its offsets in the background every auto.commit.interval.ms (default true); with it on, records processed between the last commit and a crash are read again.auto. commit
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=04.3.2" title="Kafka Simulator: EOS - offsets in a txn + fencing" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/eos-fencing/">Kafka zombie producer fencing with transactional.id</a></p>Further reading
Troubleshooting
More in Transactions
- Kafka at-most-once vs at-least-once vs exactly-once
- At-least-once delivery in Kafka: where duplicates come from
- Kafka transaction commit vs abort with read_committed
- Kafka isolation.level: read_committed vs read_uncommitted
- Kafka last stable offset (LSO) with concurrent transactions
- Kafka transaction.timeout.ms: the coordinator aborts
- Kafka read_committed during a leader election
- Kafka consume-process-produce loop with exactly-once
- Kafka processing.guarantee=exactly_once_v2 end to end