---
title: "Kafka exactly-once pipeline that survives a region failover"
description: "One Kafka transaction commits settlements and input offsets together, read_committed hides open ones, and in this run the MM2 failover loses or repeats nothing."
url: https://monedula.dev/kafka-simulator/scenarios/exactly-once-payments-pipeline/
---

# Kafka exactly-once pipeline that survives a region failover

Scenario 12.0.1 · Capstone: exactly-once payments pipeline survives a region loss. Capstones · Advanced · about 5 min · Apache Kafka 4.3.

End-to-end EOS: consume-process-produce commits input offsets + output atomically, a read_committed consumer only sees committed settlements, MM2 mirrors them to DR, and a failover resumes the downstream at its committed offset - no loss, no duplicates.

Run it: https://monedula.dev/kafka-simulator/playground/?scenario=12.0.1

## What you start with

An end-to-end exactly-once payments pipeline. A consume-process-produce loop reads `payments-in`, writes the settled record to `payments-out`, and commits the input offsets together with the output in ONE transaction; an idempotent acks=all producer deduplicates retries; a read_committed downstream consumer sees a settlement only after the transaction commits; and MirrorMaker 2 mirrors the committed output to a DR region. Watch a region failover resume the downstream at its committed offset.

## Step by step

1. **Cluster ready - EOS pipeline + DR mirror** (t+0.000s): Two regions: primary serves payments-in/payments-out; dr is the mirror target. `tx` is the idempotent+transactional pipeline producer (epoch 1); `pipeline` reads the input with offsets committed via the txn; `settle` reads the output read_committed; MM2 mirrors payments-out to dr. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.1&t=0)
2. **Input seeded · pipeline polls** (t+0.500s): ingest writes pay-1/pay-2 to payments-in (plain, immediately committed). The pipeline consumer polls them - the records to be processed-and-settled. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.1&t=1)
3. **Output on the log - uncommitted** (t+2.200s): `tx` has produced settled-1/settled-2 to payments-out (the partition HW advanced), but they belong to an OPEN transaction. `settle` (read_committed) stays at offset 0 - it cannot read past the last stable offset, so uncommitted settlements are invisible downstream. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.1&t=2)
4. **Input offsets staged in the txn** (t+2.500s): send_offsets_to_txn adds the pipeline group's input offsets to the SAME transaction as the output records - so the input is marked consumed and the output is produced atomically (consume-process-produce). [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.1&t=3)
5. **Commit - output + input offsets atomic** (t+3.400s): commit_transaction lands both the output records and the input-offset commit together. Now read_committed `settle` can see settled-1/settled-2: exactly-once, no duplicates, the input marked done in the same unit. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.1&t=4)
6. **Settlements mirrored to dr · settle drains** (t+4.000s): MM2 has mirrored the committed payments-out into the dr region (primary.payments-out) and `settle` is draining the exactly-once stream on the primary. The mirror keeps dr at RPO 0. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.1&t=5)
7. **Failover to dr - settle resumes, zero loss** (t+4.400s): The primary is lost; failover promotes dr. `settle` re-points to primary.payments-out in dr and resumes at its committed offset - it keeps reading the exactly-once settlements across the region outage with no loss and no duplicates. [Open at this step](https://monedula.dev/kafka-simulator/playground/?scenario=12.0.1&t=6)

## The takeaway

Stacking transactions, idempotence, read_committed, and a DR mirror yields end-to-end exactly-once that survives a whole-region outage - no settlement is lost and none is duplicated across the failover.

## Configs in this scenario

- [`acks`](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_acks): How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). [How acks works](https://monedula.dev/kafka-simulator/configs/acks/)

## Further reading

- [Apache Kafka documentation](https://kafka.apache.org/43/design/design/#message-delivery-semantics)