Scenario 12.0.1 · Capstone: exactly-once payments pipeline survives a region loss
Kafka exactly-once pipeline that survives a region failover
End-to-end EOS: consume-process-produce commits input offsets + output atomically, a read_committed consumer only sees committed settlements, MM2 mirrors them to DR, and a failover resumes the downstream at its committed offset - no loss, no duplicates.
Run this scenario in the playgroundCluster ready - EOS pipeline + DR mirror. t+0.000s
Two regions: primary serves payments-in/payments-out; dr is the mirror target. `tx` is the idempotent+transactional pipeline producer (epoch 1); `pipeline` reads the input with offsets committed via the txn; `settle` reads the output read_committed; MM2 mirrors payments-out to dr.
Open at this step: step 1, Cluster ready - EOS pipeline + DR mirrorInput seeded · pipeline polls. t+0.500s
ingest writes pay-1/pay-2 to payments-in (plain, immediately committed). The pipeline consumer polls them - the records to be processed-and-settled.
Open at this step: step 2, Input seeded · pipeline pollsOutput on the log - uncommitted. t+2.200s
`tx` has produced settled-1/settled-2 to payments-out (the partition HW advanced), but they belong to an OPEN transaction. `settle` (read_committed) stays at offset 0 - it cannot read past the last stable offset, so uncommitted settlements are invisible downstream.
Open at this step: step 3, Output on the log - uncommittedInput offsets staged in the txn. t+2.500s
send_offsets_to_txn adds the pipeline group's input offsets to the SAME transaction as the output records - so the input is marked consumed and the output is produced atomically (consume-process-produce).
Open at this step: step 4, Input offsets staged in the txnCommit - output + input offsets atomic. t+3.400s
commit_transaction lands both the output records and the input-offset commit together. Now read_committed `settle` can see settled-1/settled-2: exactly-once, no duplicates, the input marked done in the same unit.
Open at this step: step 5, Commit - output + input offsets atomicSettlements mirrored to dr · settle drains. t+4.000s
MM2 has mirrored the committed payments-out into the dr region (primary.payments-out) and `settle` is draining the exactly-once stream on the primary. The mirror keeps dr at RPO 0.
Open at this step: step 6, Settlements mirrored to dr · settle drainsFailover to dr - settle resumes, zero loss. t+4.400s
The primary is lost; failover promotes dr. `settle` re-points to primary.payments-out in dr and resumes at its committed offset - it keeps reading the exactly-once settlements across the region outage with no loss and no duplicates.
Open at this step: step 7, Failover to dr - settle resumes, zero lossWhat you start with
An end-to-end exactly-once payments pipeline. A consume-process-produce loop reads `payments-in`, writes the settled record to `payments-out`, and commits the input offsets together with the output in ONE transaction; an idempotent acks=all producer deduplicates retries; a read_committed downstream consumer sees a settlement only after the transaction commits; and MirrorMaker 2 mirrors the committed output to a DR region. Watch a region failover resume the downstream at its committed offset.
Step by step
- Cluster ready - EOS pipeline + DR mirror t+0.000s Two regions: primary serves payments-in/payments-out; dr is the mirror target. `tx` is the idempotent+transactional pipeline producer (epoch 1); `pipeline` reads the input with offsets committed via the txn; `settle` reads the output read_committed; MM2 mirrors payments-out to dr. Open at this step: step 1, Cluster ready - EOS pipeline + DR mirror
- Input seeded · pipeline polls t+0.500s ingest writes pay-1/pay-2 to payments-in (plain, immediately committed). The pipeline consumer polls them - the records to be processed-and-settled. Open at this step: step 2, Input seeded · pipeline polls
- Output on the log - uncommitted t+2.200s `tx` has produced settled-1/settled-2 to payments-out (the partition HW advanced), but they belong to an OPEN transaction. `settle` (read_committed) stays at offset 0 - it cannot read past the last stable offset, so uncommitted settlements are invisible downstream. Open at this step: step 3, Output on the log - uncommitted
- Input offsets staged in the txn t+2.500s send_offsets_to_txn adds the pipeline group's input offsets to the SAME transaction as the output records - so the input is marked consumed and the output is produced atomically (consume-process-produce). Open at this step: step 4, Input offsets staged in the txn
- Commit - output + input offsets atomic t+3.400s commit_transaction lands both the output records and the input-offset commit together. Now read_committed `settle` can see settled-1/settled-2: exactly-once, no duplicates, the input marked done in the same unit. Open at this step: step 5, Commit - output + input offsets atomic
- Settlements mirrored to dr · settle drains t+4.000s MM2 has mirrored the committed payments-out into the dr region (primary.payments-out) and `settle` is draining the exactly-once stream on the primary. The mirror keeps dr at RPO 0. Open at this step: step 6, Settlements mirrored to dr · settle drains
- Failover to dr - settle resumes, zero loss t+4.400s The primary is lost; failover promotes dr. `settle` re-points to primary.payments-out in dr and resumes at its committed offset - it keeps reading the exactly-once settlements across the region outage with no loss and no duplicates. Open at this step: step 7, Failover to dr - settle resumes, zero loss
The takeaway
Stacking transactions, idempotence, read_committed, and a DR mirror yields end-to-end exactly-once that survives a whole-region outage - no settlement is lost and none is duplicated across the failover.
Configs in this scenario
-
acksHow many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works
Embed this scenario
Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.
<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=12.0.1" title="Kafka Simulator: Capstone: exactly-once payments pipeline survives a region loss" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/exactly-once-payments-pipeline/">Kafka exactly-once pipeline that survives a region failover</a></p>