Resilience Patterns

Sunday 8 PM. Swiggy Checkout Spins. One Slow Service Started It.

Sunday evening, peak dinner hour. 8 PM. Swiggy's Order Service handles 80,000 concurrent checkout sessions across Bangalore, Mumbai, and Hyderabad. Somewhere in the stack, the Discount Service database starts responding in 30 seconds instead of its usual 20 milliseconds.

The Order Service calls Discount on every checkout to apply restaurant coupons and platform offers. It waits. 30 seconds. The HTTP client thread is blocked. Another checkout request arrives. Another thread blocks. Then another. Then another.

After 90 seconds, all 500 of the Order Service's Tomcat worker threads are blocked, waiting on Discount. New checkout requests sit in the accept queue. The queue fills. Checkout is effectively down.

Payment Service tries to call Order Service to confirm order details before hitting the UPI rail. Order Service cannot respond because every thread is stuck. Now Payment threads block too.

One slow Postgres instance behind Discount has taken down checkout, order confirmation, and payment initiation. The Swiggy app shows a spinner. It stays that way for 40 minutes while engineers trace thread dumps and wonder why Payment looks healthy in the pod list but fails every call.

This is a cascading failure. One sick downstream can starve threads and take healthy services with it.

Resilience patterns (circuit breaker, bulkhead, timeouts, retries done right) are how you keep blast radius small. When you finish, you can name which pattern stops which failure and what tradeoff each one buys.

Why Should You Care?

1. Interview staple at Swiggy, PhonePe, Flipkart, Razorpay. "Your dependency is slow, not down. What do you do?" Weak answer: "Add a timeout." Strong answer: breaker states, bulkhead isolation, retry policy that does not amplify load, fallback hierarchy, tuning knobs.
2. Revenue paths depend on partial failure tolerance. Checkout, UPI collect, wallet load. Users do not care which microservice broke. They care whether money moved and food arrived.
3. Microservices multiply failure surface. Ten services means ten downstreams that can drag you down. Resilience patterns are how you keep blast radius small enough to survive Sunday peak without a company-wide war room.

🟢 The Simple Version (Start Here If You're New)

The MCB on Your Flat's Switchboard

Your home has a miniature circuit breaker (MCB) on the main board. If too much current flows (short in the kitchen geyser, overloaded AC on the same phase) the MCB trips. Power to that circuit cuts immediately. The rest of the flat keeps working. Lights in the bedroom stay on.

You fix the fault, flip the MCB back. If the short is still there, it trips again. Only when the fault clears does power restore safely.

In plain words: a circuit breaker (a guard between your service and a downstream) watches calls to one dependency. When too many fail or run too slow, it trips. It stops calling that dependency and returns a fallback fast.

After a wait period, the breaker tries a few test calls in HALF-OPEN (cautious probe mode). Success closes the circuit. Failure opens it again.

Why "Just Add a Timeout" Is Incomplete Advice

In plain words: a timeout (a max wait per call) caps how long one request blocks. Standard advice: set a 2-second client timeout on Discount calls.

Timeouts help. They prevent infinite blocking. But they do not prevent pool exhaustion during a storm.

If Discount is degraded, every call may wait the full 2 seconds. Five hundred concurrent checkouts means five hundred blocked threads. The pool is still empty. New requests queue and fail.

A circuit breaker in OPEN state returns immediately. No network wait. The thread goes back to the pool in under a millisecond. Timeout bounds one call. A breaker stops calling a sick dependency altogether.

In plain words: CLOSED means healthy. Calls pass through. The breaker counts outcomes.

In plain words: OPEN means tripped. Calls are blocked. Fallback or fast error runs. The downstream gets breathing room.

In plain words: HALF-OPEN means cautious probe. Limited test traffic. The breaker decides close or re-open.

Bulkhead: One Slow Sink Should Not Flood the Whole Flat

Ships divide the hull into watertight compartments. One compartment floods. Others stay dry. The ship stays afloat.

In plain words: a bulkhead (a separate thread or connection pool per downstream) gives each dependency its own pool. Discount slowness fills only Discount's pool. Payment keeps its own threads.

The MCB analogy covers circuit breakers. Bulkheads limit damage while the breaker is still counting. You usually want both on checkout paths.

The analogy gave you the shape. Next come the YAML knobs you actually tune in production.

🟡 Going Deeper: How Breakers Actually Work

Configuration Parameters You Will Tune in YAML

These are Resilience4j names. Hystrix, Istio outlier detection, and Envoy have equivalents. The concepts transfer.

Failure rate threshold: Trip OPEN when X% of recent calls fail. Example: 50% of last 10 calls.

Slow call threshold: Treat calls over Y milliseconds as failures even if HTTP status is 200. A service answering in 8 seconds is as dangerous as one returning 500.

Slow call rate threshold: Trip when Z% of recent calls are slow, not just hard errors.

Sliding window size: How many recent calls form the decision window. Small window reacts fast but noisy. Large window is stable but slow to trip.

Wait duration in OPEN: How long before HALF-OPEN probe. Typical 30-60 seconds for external bank APIs. Shorter for internal caches.

Permitted calls in HALF-OPEN: How many test requests before close or re-open. Usually 3-10.

resilience4j:
  circuitbreaker:
    instances:
      discountService:
        failureRateThreshold: 50
        slowCallDurationThreshold: 2s
        slowCallRateThreshold: 50
        waitDurationInOpenState: 30s
        permittedNumberOfCallsInHalfOpenState: 5
        slidingWindowSize: 10

With this config: if half of the last 10 calls fail OR half are slower than 2 seconds, breaker opens. Stays open 30 seconds. Allows 5 probe calls. Clean probes close the circuit.

Timeouts vs Circuit Breaker: Different Jobs

Rule of thumb I use on fintech paths: timeout = 2-5x downstream p99 (tight, not 30s). Breaker window sized so trip happens before pool is fully drained. Bulkhead pool sized so one dependency cannot consume more than 30-40% of total outbound capacity.

PhonePe's payment path might use 800ms timeout to Bank API (p99 around 200ms), breaker on 50% failure in 20-call window, dedicated 80-thread bulkhead for bank calls separate from notification calls. Notification can lag. Bank call cannot block wallet balance read.

Cascading Failure: The Thread Pool Story

Walk through the Swiggy timeline with numbers:

1. Discount p99 goes from 20ms to 3s (still "up" by health check, returns 200).
2. Order Service has 500 threads, 2s timeout, no slow-call breaker config.
3. At 8:00:00 PM, 400 concurrent checkouts. Each thread blocks ~2s. Pool mostly full.
4. At 8:00:05 PM, 500 threads blocked. New requests wait in queue.
5. At 8:00:10 PM, Payment Service calls Order. Order cannot accept work. Payment threads block.
6. At 8:00:15 PM, on-call sees Payment pods "healthy" but error rate 100%.

A circuit breaker with slow call detection would have tripped when half of the last 10 Discount calls exceeded 2 seconds. Subsequent checkouts get cached discount fallback in 5ms. Threads never enter the 2s wait spiral.

Retries and Circuit Breakers: Partners That Can Fight Each Other

Retries help when failure is transient: packet loss, brief GC pause, one pod restarting during rolling deploy.

Retries hurt when failure is sustained: database lock storm, thread pool exhaustion downstream, dependency at capacity.

Naive retry: 3 retries, fixed 1 second gap, no jitter, retries even when breaker is OPEN.

What goes wrong at IPL payment peak:

• 50,000 PhonePe clients hit overloaded Bank API at 7:59:00 PM.
• All fail within 500ms.
• All retry at exactly 8:00:00 PM (fixed 1s backoff).
• Bank API, already at 95% CPU, gets a synchronized second wave.
• Outage extends. Breaker on each client would have stopped after window filled.

Combined pattern that survives production:

1. Retry only on idempotent operations (GET, or POST with idempotency key already stored).
2. Cap retries low: 2-3 max on user-facing sync path. More on async workers.
3. Exponential backoff with jitter: 200ms, 400ms, 800ms plus random ±30%. Spreads retry spikes.
4. Retry inside CLOSED/HALF-OPEN only. When breaker is OPEN, skip retries entirely. Go straight to fallback. Retrying an open circuit wastes threads and hammers a dependency you already decided is sick.
5. Count retry failures toward breaker threshold. All 3 failed attempts in one user request = 3 failures in the sliding window. Trip faster during real outages.

// Illustrative pattern: retry respects breaker state
async function callBankWithResilience(req: PaymentReq): Promise<BankResult> {
 const breaker = registry.circuitBreaker('bankApi');
 if (breaker.state === 'OPEN') {
   return bankFallback(req); // no retry, no blocking wait
 }
 return retry(
   () => breaker.execute(() => bankClient.charge(req)),
   { maxAttempts: 3, backoff: expJitter(200), retryOn: isTransient }
 ).catch(() => bankFallback(req));
}

Tradeoff table for retries on sync checkout:

I bias toward fewer retries on the browser-facing path and more on the async settlement worker that can wait 30 seconds without a human staring at a spinner.

Configuration gets you deployed. The Architect's Corner is where I tell you what still breaks on Sunday peak.

🔴 Architect's Corner

Bulkhead: Contain Flooding to One Compartment

Ship bulkheads: watertight walls divide the hull. One compartment floods, others stay dry. Ship stays afloat.

Software bulkhead: separate thread pool (or connection pool, or semaphore) per downstream dependency.

Without bulkheads, Discount slowness consumes threads that Payment and Catalog also need. Payment is healthy. Discount is not. Payment still fails because it cannot get a thread.

With bulkheads, Discount gets 40 threads max. Payment gets its own 40. Discount pool fills and rejects fast. Payment pool untouched. Catalog keeps serving menu enrichment.

Sizing bulkheads is a tradeoff, not a formula:

• Too small: False rejections under normal Sunday peak. You shed healthy traffic.
• Too large: Behaves like shared pool. Defeats the purpose.
• Practical start: Allocate per dependency based on peak QPS x p99 latency x safety factor. Revisit after Diwali sale load test.

Relationship to circuit breaker: bulkhead limits damage while breaker is counting toward trip. Breaker limits damage after trip. I have seen teams deploy breakers without bulkheads and still lose the fleet in the 10 failures it takes to open the circuit.

Fallback Hierarchy: Open Circuit Is Not User-Facing Failure

When Discount breaker opens on Swiggy, the worst outcome is a broken checkout page. The acceptable outcome is checkout without dynamic coupon. The great outcome is stale-but-reasonable coupons from Redis.

Fallback ladder for Discount:

1. Redis cache of restaurant-level offers (may be 5 minutes stale).
2. Platform-wide default: 0% extra discount, checkout proceeds at menu price.
3. UI banner: "Live offers unavailable. You can still place your order."
4. Hard error only if cart cannot be priced at all.

Each step preserves more revenue than the one below.

PhonePe Bank API breaker open:

1. Queue payment for async retry if business rules allow (bill pay, not in-store QR).
2. Offer UPI collect via alternate rail if redundancy exists.
3. Clear message: "Bank partner busy. Try again in 2 minutes." with idempotent resume token.
4. Never silent double charge. Idempotency key on every attempt.

Resilience4j wiring:

@CircuitBreaker(name = "discountService", fallbackMethod = "discountFallback")
public DiscountResult applyDiscount(Cart cart) { ... }

private DiscountResult discountFallback(Cart cart, Throwable t) {
 return cache.get(cart.restaurantId())
     .orElse(DiscountResult.none());
}

Fallback methods must be fast and must not call the same sick dependency. I have debugged a "fallback" that called Discount again "just to try cache refresh." That is not a fallback. That is another outage.

Netflix Hystrix to Resilience4j

Hystrix (@HystrixCommand) introduced circuit breakers to mainstream Java microservices. Maintenance mode since 2018. New Spring Boot work should use Resilience4j.

Resilience4j modules: circuit breaker, retry, bulkhead, rate limiter, time limiter. Integrates with Spring Boot Actuator for /actuator/circuitbreakers metrics. Compose annotations or functional style.

What marketing slides skip: libraries do not choose your thresholds. You do. Wrong thresholds are worse than no breaker (flapping OPEN/HALF-OPEN, or never tripping during slow-death).

Chaos Engineering: Prove It Before Sunday Peak

Unit tests verify breaker math. They do not verify your fallback returns legal prices, or that bulkhead pool sizes match real traffic mix, or that retries stop when OPEN.

Mature teams inject failure in staging (sometimes prod with blast radius controls):

• Latency injection: Discount responds in 5s.
• Error injection: 50% 503 from mock Bank API.
• Process kill: terminate Discount pod mid-checkout load test.

Netflix Chaos Monkey kills random instances. Swiggy-style chaos verifies breaker trips within N seconds, fallback SLO holds, no thread pool above 80% for more than 30s.

Minimum chaos checklist before declaring resilience "done":

• Slow downstream (3s latency) does not raise Order error rate above 1% with fallback.
• Breaker OPEN visible on dashboard within expected window.
• HALF-OPEN closes cleanly when dependency recovers.
• Retry storm test: fixed backoff without jitter fails load test. With jitter, passes.

How India-Scale Systems Apply These Patterns

Swiggy (food delivery peak): Checkout path uses breakers on Discount, Payment orchestration, and address validation. Fallback discounts from cache. Bulkheads separate "read menu" from "write order" outbound pools. Sunday 8 PM is the load test nature runs every week.

PhonePe (UPI volume): Bank connector breakers with conservative OPEN wait (banks recover slowly). Retries only with idempotency keys. Sync user-facing path fails to clear message rather than hanging 30s. Async reconciliation workers retry with longer backoff.

Flipkart (Big Billion Days): Catalog and inventory read paths get aggressive bulkheads so write-heavy checkout does not starve browse. Breakers on recommendation service with static fallback (bestseller list) when personalization is down.

Hotstar (live stream spikes): Breakers on ad-personalization and secondary metadata. Core video path isolated. You watch the match even when coupon overlay fails.

Zomato vs Swiggy engineer debate (real tension): "OPEN wait should be 5s so users get discounts back quickly." Counter: "Discount DB recovery takes 20-30s after connection pool clear. 5s causes thrashing. Total outage window grows." Both care about UX. The senior answer is measure recovery time, tune wait to p95 recovery, accept stale fallback during window.

The Decision Matrix

Tuning Tradeoffs (The Knobs That Start Fights)

There is no universal correct row. PhonePe bank path is conservative on OPEN wait. Swiggy Discount can be more aggressive because fallback to zero discount is cheap.

Common Mistakes

"We set a 30-second timeout, we are safe." Thirty seconds on a call that normally takes 20ms means each failing call holds a thread for 30 seconds. Pool empties in one traffic slice. Set timeout to a small multiple of p99.

"Circuit breaker opened, mission accomplished." OPEN without fallback still returns errors to users. Breaker contains blast radius. Fallback preserves business function. Configure both.

"Retries always improve reliability." Retries on non-idempotent POST without keys cause duplicate orders. Retries during sustained outage amplify load. Cap retries. Add jitter. Respect OPEN state.

"Breaker only counts HTTP 5xx." Slow 200 responses are the most common production killer. Enable slow call rate thresholds.

"We have a breaker, skip bulkhead." The failures that trip the breaker already consumed threads. Bulkhead limits pre-trip damage. Deploy both on revenue paths.

"Breaker state is only for debugging." Expose OPEN/HALF-OPEN on dashboards and alerts. On-call must distinguish "protecting us" from "dependency healthy." Otherwise you page for the cure, not the disease.

🧠 Key Takeaways

Circuit breaker sits between caller and downstream, tracks failures and slow calls, trips to OPEN to fail fast and free threads, probes recovery in HALF-OPEN.

Three states: CLOSED (normal), OPEN (block calls, run fallback), HALF-OPEN (limited test traffic).

Timeouts bound single-call wait. Breakers stop calling a sick dependency at scale. You need both.

Bulkheads isolate pools per downstream so one slow service cannot starve threads for healthy services.

Retries pair with exponential backoff and jitter. Do not retry through an OPEN breaker. Count retry failures toward trip threshold.

Fallback hierarchy degrades gracefully: cache → safe default → soft UX → hard error last.

Resilience4j is the modern Java default (replacing Hystrix). Thresholds in YAML, metrics via Actuator.

Chaos engineering validates patterns under real failure. Config correctness is not behavior correctness.

Think About It

PhonePe Payment Service calls four downstreams: Bank API, Fraud Scoring, Notification, Audit Log. Bank API has a breaker with bank-specific fallback. Audit Log is down. Should payment still succeed? What is your compensating action if audit must be durable for RBI compliance?

Swiggy Order Service breaker on Discount uses 30s OPEN wait. A product manager wants 5s because "users miss coupons." An SRE cites recovery metrics showing 25s p95 database recovery. How do you settle the argument with data?

You have breakers but no bulkheads. Breaker trips after 10 failures. By trip time, 400 threads were already blocked in slow Discount calls. What does this teach about ordering resilience patterns on a shared Tomcat pool?

Further Reading

• Michael Nygard, Release It! (circuit breaker, bulkhead, timeout, stability patterns)
• Resilience4j docs (configuration and Spring Boot integration)
• Netflix Hystrix wiki (historical, still good for mental model)
• Casey Rosenthal & Nora Jones, Chaos Engineering (failure injection practice)
• AWS re:Invent talks on resilience and bulkhead isolation in large microservice fleets