The Desi Architect
Notes / Docker Sandboxes / Production Chaos Game Day: Failure Injection & Incident Recovery
Docker Sandboxes · Tier 4: Architect & Chaos · Lab 59

Production Chaos Game Day: Failure Injection & Incident Recovery

Put on your SRE on-call pager and run controlled chaos drills. Inject latency spikes, saturate connection pools, evict container pods, diagnose root causes from live Prometheus metrics, and execute runbooks to restore healthy traffic.

Time30 minutes
StackChaos EngineeringPorts 18130
MemoryAbout 200 MB
Before thisLab 43Circuit Breaker & Thread Bulkhead
Download lab (.zip) View steps
Verified runWindows, Mac, LinuxBash and PowerShell
  1. 1PrepareWhat you will learn
  2. 2PredictGuess before you run
  3. 3RunStep by step
  4. 4ObserveWhat you should see
  5. 5Break itChange one thing
  6. 6Clean upLeave nothing running
On this page
  1. 00 / What You Will Learn
  2. 01 / Why It Matters
  3. 02 / What We Are Building
  4. 03 / Words You Need
  5. 04 / Predict First
  6. 05 / Step by Step
  7. 06 / What You Should See
  8. 07 / Look Inside
  9. 08 / Break It, Fix It
  10. 09 / In Real Projects
  11. 10 / Interview Corner
  12. 11 / Clean Up

00 /What You Will Learn

By the end you will be able to
  • Conduct controlled Game Day chaos experiments on live microservice topologies.
  • Reproduce the negative control: cascading platform failure caused by unmitigated downstream latency and pool starvation.
  • Isolate failure blast radius using circuit breakers that fast-fail with graceful degraded fallbacks in under 15 ms.
  • Resolve connection pool starvation using dynamic capacity expansion and query deadlines.
  • Test container orchestrator self-healing and traffic redirection during sudden worker pod evictions.
  • Scrape and inspect live metrics in Prometheus text exposition format.
  • Measure Mean Time To Recovery (MTTR) against strict service level objectives (SLOs).

Before you start

Finish Lab 43 (Circuit Breaker & Thread Bulkhead Isolation) and Lab 47 (OpenTelemetry Tracing) first. You should understand latency percentiles, cascading timeouts, thread bulkheads, and Prometheus metrics. Docker Desktop must be running.