Scenario 03.2.5 · Returning replica and log truncation

Kafka log truncation when an old leader returns

Replication · Advanced · about 4 min · Apache Kafka 4.3

A replica holding records the new leader never got truncates them at the divergence point (found via leader epochs) before it can rejoin the ISR - the lossless guard against divergent logs.

What you start with

A replica may hold records BEYOND the high watermark - uncommitted writes the leader appended but the ISR never fetched. Here both followers are throttled so O3,O4 land only on the leader; the leader is then isolated and a follower at the HW is elected, orphaning that uncommitted tail.

Step by step

1. Three replicas in sync t+0.000s RF=3, min.insync.replicas=2. broker-1 leads; broker-2 and broker-3 are followers. All three are in the ISR at LEO 0 - every replica holds an identical, empty log. Open at this step: step 1, Three replicas in sync
2. O1, O2 committed (HW=2) t+5.000s Two records replicate to all three replicas, so the high watermark advances to 2 and the acks=all producer is acked. HW=2 is the durability frontier: only offsets below it are committed and readable. Open at this step: step 2, O1, O2 committed (HW=2)
3. O3 lands on the leader only t+9.000s broker-2 and broker-3 are throttled (frozen at LEO 2), so O3 reaches only the leader: broker-1 LEO climbs to 3 while the HW stays pinned at 2. The record is uncommitted - and acks=all leaves the producer waiting, never telling it the write is durable. Open at this step: step 3, O3 lands on the leader only
4. Leader holds an uncommitted tail t+11.000s O4 also lands on broker-1 alone (LEO 4). The leader now carries two records - offsets 2 and 3 - that no other in-sync replica has and that were never committed. This is the divergence hazard: a replica ahead of the high watermark. Open at this step: step 4, Leader holds an uncommitted tail
5. Partition isolates broker-1 t+14.000s A network partition isolates broker-1. Once it loses its controller session, the controller fences it and elects broker-2, the first remaining ISR replica (LEO 2, leader epoch bumps to 1), and broker-1 drops out of the ISR - taking its uncommitted offsets 2,3 with it. Open at this step: step 5, Partition isolates broker-1
6. Return → log truncation t+18.000s The network heals. broker-1 returns with LEO 4, ahead of the leader's LEO 2 - a divergent log. Using leader epochs it learns epoch 0 ended at offset 2 on the new leader and TRUNCATES there (offsets 2,3 discarded: the log_truncate moment), then refetches from the leader. Open at this step: step 6, Return → log truncation
7. Rejoined, logs consistent t+22.000s broker-1 has caught up to LEO 2 and is back in the ISR. Every replica holds the same committed log again. No committed record was ever lost - truncation only dropped writes that never crossed the high watermark, which is exactly why they were safe to drop. Open at this step: step 7, Rejoined, logs consistent

The takeaway

When the old leader returns it truncates its log where it diverges from the new leader (found via leader epochs; here offset 2, the HW) before rejoining the ISR - only never-committed records are dropped, so the cluster reconverges without losing a single acked write.

Configs in this scenario

• min.insync.replicas The fewest in-sync replicas, leader included, that must have a write for an acks=all produce to succeed (default 1); with fewer in the ISR the leader rejects the write with NOT_ENOUGH_REPLICAS. How min.insync.replicas works
• acks How many replicas must have a write before the producer counts it as sent: 0 (none), 1 (the leader only) or all (every in-sync replica, the default, and required for idempotence). How acks works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=03.2.5" title="Kafka Simulator: Returning replica and log truncation" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/returning-replica-truncation/">Kafka log truncation when an old leader returns</a></p>

Further reading

Apache Kafka documentation

More in Replication

• Kafka ISR (in-sync replicas): how replicas leave and rejoin
• min.insync.replicas with acks=all: when Kafka refuses writes
• Kafka slow follower: replica.lag.time.max.ms and the ISR
• Kafka high watermark pinned by a slow in-sync replica
• Losing two Kafka brokers with min.insync.replicas=2
• Kafka preferred leader election after a rack failure
• Kafka unclean.leader.election.enable: offline or data loss
• Kafka unclean leader election: permanent loss of acked data
• Kafka broker.rack: rack-aware replica placement
• Kafka rack-aware reassignment after adding a rack
• Increase the replication factor of an existing Kafka topic

All scenarios