The Desi Architect
Notes / Docker Sandboxes / Kafka Consumer Group Partition Rebalancing and Pod Eviction
Docker Sandboxes · Tier 3: Reliability & Messaging · Lab 38

Kafka Consumer Group Partition Rebalancing and Pod Eviction

Watch Kafka move partitions between consumers when one joins and when one is killed with no goodbye. Time how long a dead worker's partition sits unread, and check that no order was lost.

Time30 minutes
StackPython 3Ports 19092, 18090
MemoryAbout 1 GB
Before thisLab 37Kafka Consumer Groups, Offset Comm
Download lab (.zip) View steps
Verified runWindows, Mac, LinuxBash and PowerShell
  1. 1PrepareWhat you will learn
  2. 2PredictGuess before you run
  3. 3RunStep by step
  4. 4ObserveWhat you should see
  5. 5Break itChange one thing
  6. 6Clean upLeave nothing running
On this page
  1. 00 / What You Will Learn
  2. 01 / Why It Matters
  3. 02 / What We Are Building
  4. 03 / Words You Need
  5. 04 / Predict First
  6. 05 / Step by Step
  7. 06 / What You Should See
  8. 07 / Look Inside
  9. 08 / Break It, Fix It
  10. 09 / In Real Projects
  11. 10 / Interview Corner
  12. 11 / Clean Up

00 /What You Will Learn

By the end you will be able to
  • Explain what a rebalance is and what triggers one.
  • Explain why a crash is noticed only after the session timeout.
  • Say what happens to the messages waiting on a dead worker's partition.
  • Explain why a rebalance can cause repeated deliveries.

Before you start

Finish Lab 37 (Consumer Groups) first. Docker Desktop with about 2 GB of RAM for Docker, and Python 3.