Scenario 02.3.5 · Evicted-member ghost & survivor takeover

Evicted Kafka consumer: how a survivor takes its partition

Consumer groups · Intermediate · about 3 min · Apache Kafka 4.3

A crashed member is evicted at session.timeout.ms but lingers as a dimmed "evicted" ghost row while a survivor inherits its partition via rebalance.

What you start with

When a crashed member is evicted at session.timeout.ms, its partitions are reassigned to a survivor via rebalance. This scenario keeps the evicted member visible as a dimmed "evicted" ghost row so you can watch the handover from old owner to new.

Step by step

1. Three-member group ready t+0.000s A 3-member cooperative-sticky group at rest: c1 owns p0, c2 owns p1, c3 owns p2. Every partition has exactly one owner. Open at this step: step 1, Three-member group ready
2. All consuming · lag 0 t+0.900s The producer wrote one record to each partition and all three members polled them - every position equals the high-watermark, so group lag is 0. Open at this step: step 2, All consuming · lag 0
3. c2 crashed · p1 stalls t+2.500s c2 was killed (kill_consumer) at t=1000 but is still a member that OWNS p1 - its missed heartbeats have not yet crossed session.timeout.ms. p1 stalls (position 1, HW 2 → lag 1) while c1 and c3 keep draining p0/p2. The dead owner blocks its partition. Open at this step: step 3, c2 crashed · p1 stalls
4. Evicted ghost · survivor took p1 t+3.400s At t=3000 (kill + session.timeout.ms, 2000 ms here; real brokers require at least 6 s) the coordinator evicts c2. It is removed from the group; the simulator keeps it on screen as a dimmed evicted ghost row so you can still see who left. The incremental rebalance hands its partition p1 to survivor c1 (now owns p0+p1). Open at this step: step 4, Evicted ghost · survivor took p1
5. Backlog drained · ghost lingers t+4.500s c1 has polled the p1 backlog so every position is caught up to the HW again - lag is 0 and the group is healthy on two members. The ghost c2 is a simulator display only (kafka-consumer-groups would list just c1 and c3); it clears when that id rejoins. Open at this step: step 5, Backlog drained · ghost lingers

The takeaway

After eviction the dead member's partitions move to a survivor; the lingering ghost row makes clear the partition keeps being served even though its original owner is gone.

Configs in this scenario

• session.timeout.ms How long the group coordinator waits for a heartbeat from a classic-protocol consumer before it removes the member and rebalances (default 45000, 45 seconds); with group.protocol=consumer the broker group.consumer.session.timeout.ms applies instead. How session.timeout.ms works

Embed this scenario

Paste this into a blog post, a course page or your team's docs. The scenario plays inside your page, and the link under it points readers to this walkthrough.

<iframe src="https://monedula.dev/kafka-simulator/embed/?scenario=02.3.5" title="Kafka Simulator: Evicted-member ghost &amp; survivor takeover" width="100%" height="520" style="border:1px solid #d0d0d0;border-radius:8px" loading="lazy"></iframe>
<p><a href="https://monedula.dev/kafka-simulator/scenarios/evicted-member-ghost/">Evicted Kafka consumer: how a survivor takes its partition</a></p>

Further reading

Apache Kafka documentation

More in Consumer groups

• Kafka consumer poll loop: subscribe, poll, fetch
• Kafka auto.offset.reset: earliest vs latest vs none
• Kafka committed offsets: enable.auto.commit and restart
• Replay a Kafka topic: consumed records stay in the log
• Kafka consumer group: how partitions split across members
• Kafka consumer group rebalance when a consumer joins
• Kafka partition assignors: range vs round-robin vs sticky
• Kafka eager vs cooperative rebalancing (cooperative-sticky)
• Kafka consumer crash and session.timeout.ms
• Kafka max.poll.interval.ms: evicting a stuck consumer
• Kafka static membership: group.instance.id avoids rebalances
• Kafka consumer lag: why it grows and how it drains

All scenarios