Circuit Breakers

Sunday 8 PM. Swiggy Goes Down. One Slow Service Is the Reason.

Sunday evening, peak dinner hour. 8 PM. Swiggy's servers handle 80,000 concurrent checkout sessions across Bangalore, Mumbai, and Hyderabad. Somewhere in the stack, the Discount Service's database starts responding in 30 seconds instead of its usual 20 milliseconds.

The Order Service calls the Discount Service for every checkout. It waits. 30 seconds. The HTTP client thread is blocked. Another checkout request comes in. Another thread blocks. Then another. Then another.

After 90 seconds, all 500 of the Order Service's thread pool workers are blocked, waiting on the Discount Service. New checkout requests queue up. The queue fills. Checkout is effectively down.

The Payment Service tries to call the Order Service to confirm order details before hitting the UPI rail. The Order Service's threads are all blocked, so it cannot respond. Now the Payment Service's threads block waiting on the Order Service.

One slow database in the Discount Service has taken down checkout, order management, and payment confirmation. Swiggy's checkout page shows a spinner. It stays that way for 40 minutes while engineers scramble to figure out what's wrong.

This is a cascading failure. One slow dependency can drain an entire thread pool.

A circuit breaker stops calling a sick downstream once the pattern is clear. When you finish, you can explain CLOSED, OPEN, and HALF-OPEN and why timeouts alone are not enough.

Why Should You Care?

1. Cascading failure is the default outcome without containment. Distributed systems fail partially. One service degrades. The question is not whether this happens (it will) but whether the degradation spreads to take down healthy services that had nothing to do with the original failure. Circuit breakers are how you draw a hard line around a sick dependency.
2. Interview staple at Swiggy, PhonePe, Flipkart, and Razorpay. "Your downstream is slow, not down. What do you do?" Weak answer: "Add a timeout." Strong answer: breaker states, configuration knobs, slow-call thresholds, fallback hierarchy, and how breakers interact with retries and bulkheads.
3. Revenue paths cannot afford thread pool exhaustion. Checkout, UPI collect, wallet load. Users do not care which microservice broke. They care whether money moved and food arrived. A breaker that trips OPEN and serves a fallback in 5 milliseconds beats a spinner that blocks for 30 seconds while the pool drains.

🟢 The Simple Version (Start Here If You're New)

The MCB on Your Flat's Switchboard

Your home has a miniature circuit breaker (MCB) on the main switchboard. If too much current flows (a short circuit in the kitchen geyser, an overloaded air conditioner on the same phase) the MCB trips. Power to that circuit cuts off immediately. The rest of the flat keeps working. Lights in the bedroom stay on.

You fix the problem, then manually flip the MCB back on. If the fault is still there, it trips again. Only once the fault is cleared can you restore power safely.

In plain words: a circuit breaker (a guard between your service and one downstream) works the same way. It sits between Order Service and Discount Service. It watches calls. When enough fail or time out, it trips OPEN. No more calls to Discount. Immediate fallback instead. Order threads free up. Checkout stays alive.

After a wait period, the breaker sends test calls in HALF-OPEN (cautious probe mode). Success closes the circuit. Failure opens it again.

Why Timeouts Alone Are Not Enough

In plain words: a timeout (a max wait per call) caps how long one request blocks. Set a 2-second timeout on Discount calls. If Discount does not answer in 2 seconds, fail fast.

Timeouts help. But they do not fully solve the problem.

If Discount is slow, each request still blocks for the full 2-second timeout. Five hundred concurrent requests means five hundred blocked threads. The pool is still exhausted. New requests queue and fail.

A circuit breaker notices that the last 10 calls all timed out. It opens the circuit. Later calls return immediately. No attempt to contact Discount. Threads free in microseconds, not after 2 seconds each.

Timeout bounds one call. A breaker stops calling a sick dependency once the pattern is clear.

The MCB analogy gave you the idea. Next come the three states and the knobs you tune in YAML.

🟡 Going Deeper

Three States of a Circuit Breaker

In plain words: CLOSED means healthy default. All calls pass through. The breaker tracks success and failure rates.

In plain words: OPEN means tripped. Calls are blocked. Too many failures or timeouts triggered the trip. Requests return immediately with fallback or error. No call hits the downstream. Your thread pool stays free.

In plain words: HALF-OPEN means cautious probe. After a wait (say 30 seconds), a few test requests go through. Success closes the circuit. Failure reopens it for another wait.

CLOSED means healthy traffic flows. OPEN means the caller is protecting itself and the downstream from more damage. HALF-OPEN means cautious probing: "has Discount recovered, or are we about to flood it again?"

State transition cheat sheet:

Configuration Parameters You Will Tune in Production

These are Resilience4j names. Hystrix, Istio outlier detection, and Envoy have equivalents. The concepts transfer.

Failure rate threshold: How many failures (or what failure percentage) triggers the trip to OPEN. A typical setting: trip if 50% of the last 10 calls fail.

Slow call threshold: Calls that take longer than X milliseconds are counted as failures too. A Discount Service call that takes 3 seconds is not a "failure" in the HTTP sense (no error code) but it is just as damaging to thread pool health. This is the knob most teams forget and regret.

Slow call rate threshold: Trip OPEN when Z% of recent calls are slow, not just when hard errors spike.

Sliding window size: How many recent calls form the decision window. Small window reacts fast but is noisy (one bad deploy blip trips the breaker). Large window is stable but slow to trip (more threads blocked before OPEN).

Wait duration in OPEN state: How long the breaker stays open before trying HALF-OPEN. 30-60 seconds is typical for external APIs. Too short: the breaker flips back before the downstream recovers, causing HALF-OPEN thrash. Too long: the breaker stays open even after recovery, blocking traffic unnecessarily.

HALF-OPEN permitted calls: How many test requests to allow in HALF-OPEN before deciding to close or re-open. Typically 5-10.

resilience4j:
 circuitbreaker:
   instances:
     discountService:
       failureRateThreshold: 50
       slowCallDurationThreshold: 2s
       slowCallRateThreshold: 50
       waitDurationInOpenState: 30s
       permittedNumberOfCallsInHalfOpenState: 5
       slidingWindowSize: 10

With this config, if 50% of the last 10 calls fail, or if 50% of calls take more than 2 seconds, the breaker opens. It stays open for 30 seconds, then tries 5 test calls in HALF-OPEN.

Tuning tradeoff nobody admits in the design doc:

There is no universal correct row. PhonePe's bank connector tends conservative (banks recover slowly). Swiggy's Discount breaker can be more aggressive because fallback to zero discount is cheap.

Retry With Exponential Backoff and Jitter

Retries and circuit breakers work together, but they can also fight each other. When a call fails, a simple retry might succeed if the failure was transient (a brief network blip, one pod restarting during rolling deploy). But naive retries make cascading failures worse: 10,000 clients each retry 3 times, multiplying the load on an already struggling service.

Exponential backoff: wait 200ms before first retry, 400ms before second, 800ms before third. Each retry doubles the wait.

Jitter: add a random offset to the wait (e.g., 800ms plus or minus 300ms). Without jitter, all clients that failed at the same moment retry at the same moment, creating a "thundering herd" that overwhelms the recovering service.

Combined pattern that survives Sunday peak:

1. Retry only on idempotent operations (GET, or POST with idempotency key already stored).
2. Cap retries low on user-facing sync paths: 2-3 max. More on async workers.
3. Exponential backoff with jitter on every retry.
4. Do not retry when breaker is OPEN. Go straight to fallback. Retrying an open circuit wastes threads and hammers a dependency you already decided is sick.
5. Count retry failures toward breaker threshold. Three failed attempts in one user request equals three failures in the sliding window. Trip faster during real outages.

// Illustrative: retry respects breaker state
async function callDiscountWithResilience(cart: Cart): Promise<DiscountResult> {
 const breaker = registry.circuitBreaker('discountService');
 if (breaker.state === 'OPEN') {
   return discountFallback(cart); // no retry, no blocking wait
 }
 return retry(
   () => breaker.execute(() => discountClient.apply(cart)),
   { maxAttempts: 3, backoff: expJitter(200), retryOn: isTransient }
 ).catch(() => discountFallback(cart));
}

I bias toward fewer retries on the browser-facing path and more on the async settlement worker that can wait 30 seconds without a human staring at a spinner.

States and YAML knobs get you to production. The Architect's Corner is where I walk through what still breaks on Sunday peak.

🔴 Architect's Corner

Cascading Failure: The Thread Pool Story

The Swiggy incident is not abstract. Walk through it with numbers:

1. Discount p99 goes from 20ms to 3s (still "up" by health check, returns HTTP 200).
2. Order Service has 500 threads, 2s timeout, no slow-call breaker config.
3. At 8:00:00 PM, 400 concurrent checkouts. Each thread blocks roughly 2s. Pool mostly full.
4. At 8:00:05 PM, 500 threads blocked. New requests wait in queue.
5. At 8:00:10 PM, Payment Service calls Order. Order cannot accept work. Payment threads block.
6. At 8:00:15 PM, on-call sees Payment pods "healthy" but error rate 100%.

A circuit breaker with slow call detection would have tripped when half of the last 10 Discount calls exceeded 2 seconds. Subsequent checkouts get cached discount fallback in 5ms. Threads never enter the 2s wait spiral. Payment keeps confirming orders because Order threads stay free.

What the diagram teaches: the Discount Service is the root cause, but Order and Payment are the victims. The failure propagates through blocked threads, not through error codes. Health checks on Payment pods look green while every outbound call fails.

The Bulkhead Pattern: Breaker's Partner, Not Replacement

The bulkhead pattern in ships: a vessel is divided into watertight compartments. If one compartment floods, the others stay dry. The ship does not sink.

In software, the bulkhead pattern isolates resources per downstream service. Each downstream gets its own thread pool. If the Discount Service becomes slow and exhausts its thread pool, it does not touch the thread pool used for the Payment Service or the Catalog Service.

Without bulkheads, one slow downstream can exhaust the entire shared thread pool. With bulkheads, the degradation is contained.

Relationship to circuit breaker: bulkhead limits damage while the breaker is counting toward trip. Breaker limits damage after trip. I have seen teams deploy breakers without bulkheads and still lose the fleet in the 10 failures it takes to open the circuit. By trip time, 400 threads were already blocked in slow Discount calls.

Sizing bulkheads is a tradeoff:

• Too small: false rejections under normal Sunday peak. You shed healthy traffic.
• Too large: behaves like a shared pool. Defeats the purpose.
• Practical start: allocate per dependency based on peak QPS x p99 latency x safety factor. Revisit after Diwali sale load test.

Netflix Hystrix to Resilience4j

Netflix Hystrix was the original Java circuit breaker library, built for the Netflix microservices stack. It introduced the circuit breaker pattern to mainstream Java development. If you see older Spring Boot code with @HystrixCommand, that is Hystrix.

Hystrix entered maintenance mode in 2018. The replacement for Spring Boot applications is Resilience4j.

Resilience4j is a lightweight library with modules for circuit breakers, rate limiters, retry, bulkhead, time limiters, and cache. It integrates natively with Spring Boot Actuator, exposing circuit breaker state and metrics via health endpoints. Configuration is done in application.yml (see Going Deeper section).

Resilience4j modules compose: you can wrap a call in circuit breaker + retry + bulkhead + time limiter. The order matters. Retry inside breaker on CLOSED state. Fallback when OPEN. Bulkhead limits concurrent calls regardless of state.

What marketing slides skip: libraries do not choose your thresholds. You do. Wrong thresholds are worse than no breaker (flapping OPEN/HALF-OPEN forever, or never tripping during slow-death latency).

Graceful Degradation and Fallback Hierarchy

A circuit breaker opening is not a user-visible error if you have fallbacks. The goal is graceful degradation: serve a reduced but functional experience rather than an error page.

Fallback ladder for Discount Service down on Swiggy:

1. Return cached discounts from Redis (stale data, but functional).
2. Return a default "0% discount": checkout works, user does not see dynamic pricing.
3. Show a "Discounts unavailable right now" message: checkout still works.
4. Show an error only if none of the above is possible.

Each level degrades gracefully. Most users complete their order. Some miss discounts. None face a broken checkout page.

PhonePe Bank API breaker open:

1. Queue payment for async retry if business rules allow (bill pay, not in-store QR).
2. Offer UPI collect via alternate rail if redundancy exists.
3. Clear message: "Bank partner busy. Try again in 2 minutes." with idempotent resume token.
4. Never silent double charge. Idempotency key on every attempt.

This fallback logic lives in the circuit breaker's fallback method:

@CircuitBreaker(name = "discountService", fallbackMethod = "discountFallback")
public DiscountResult applyDiscount(Cart cart) { ... }

private DiscountResult discountFallback(Cart cart, Throwable t) {
 return cache.get(cart.restaurantId())
     .orElse(DiscountResult.none());
}

Fallback methods must be fast and must not call the same sick dependency. I have debugged a "fallback" that called Discount again "just to try cache refresh." That is not a fallback. That is another outage.

Chaos Engineering: Breaking Things on Purpose

Swiggy, Flipkart, and other mature engineering teams practice chaos engineering: intentionally injecting failures into staging (and sometimes production) to verify that circuit breakers, bulkheads, and fallbacks work as expected.

Netflix's Chaos Monkey terminates random production instances. Swiggy-style chaos engineering injects latency into specific service calls and verifies that circuit breakers trip within expected thresholds and that fallbacks serve correct results.

Unit tests verify breaker math. They do not verify your fallback returns legal prices, or that HALF-OPEN closes cleanly when the dependency recovers, or that retries stop when OPEN.

Minimum chaos checklist before declaring circuit breakers "done":

• Slow downstream (3s latency) does not raise Order error rate above 1% with fallback.
• Breaker OPEN visible on dashboard within expected window.
• HALF-OPEN closes cleanly when dependency recovers.
• Retry storm test: fixed backoff without jitter fails load test. With jitter, passes.

How India-scale systems do it

Swiggy: Checkout and Discount Containment

Sunday 8 PM is the load test nature runs every week. Checkout path uses circuit breakers on Discount Service, payment orchestration, and address validation. When Discount trips OPEN, fallback serves cached restaurant-level offers from Redis. Zero discount is acceptable. Broken checkout is not.

Tradeoff: stale coupons for 30-60 seconds during OPEN window costs some margin. Cheaper than 40 minutes of checkout down and Twitter outrage.

Breaker tuning on Discount tends aggressive on slow-call threshold (trip when p99 blows past 2s) because fallback is cheap. OPEN wait is tuned to measured database recovery time, not product manager gut feel.

PhonePe: Bank Connector Conservatism

UPI volume means every millisecond on the sync path matters. Bank API circuit breakers use conservative OPEN wait (banks recover slowly, often 20-30 seconds after connection pool clears). Retries only with idempotency keys. When breaker is OPEN, sync user-facing path fails to a clear message rather than hanging 30 seconds.

Tradeoff: users see "bank busy" more often than engineers would like. Alternative (long hang) causes thread exhaustion across payment pods during IPL payment peak.

Async reconciliation workers retry with longer backoff. The breaker protects the hot path. The worker heals the ledger.

Flipkart: Big Billion Days Isolation

During BBD, catalog browse and checkout write paths get separate bulkheads and independent breakers. Recommendation service breaker opens to static bestseller fallback when personalization is down. Users still find products. Checkout breakers on inventory reservation service prevent one slow shard from blocking the entire cart.

Tradeoff: personalized recommendations disappear during partial outage. Revenue from browse-to-cart conversion still flows.

Razorpay: Merchant API Protection

When a downstream bank partner degrades, Razorpay's payment orchestration layer trips breakers per rail, not globally. One sick HDFC connector does not block ICICI traffic. Fallback routes to alternate rail where merchant config allows.

Tradeoff: routing complexity and reconciliation overhead. Merchants must configure fallback rails in advance, not during outage.

Hotstar: Core Path vs Enrichment

Live IPL match streaming isolates the video delivery path from ad personalization and secondary metadata services. Breakers on enrichment services fail silently (no coupon overlay, generic ad pod). Core video path never calls enrichment synchronously on the critical render chain.

Tradeoff: ad revenue dips during personalization outage. Match keeps playing. That is the correct priority order.

Zomato vs Swiggy: The 5-Second vs 30-Second OPEN Wait Debate

Real tension between product and SRE teams: "OPEN wait should be 5 seconds so users get discounts back quickly." Counter: "Discount DB recovery takes 20-30 seconds after connection pool clear. 5 seconds causes HALF-OPEN thrash. Total outage window grows."

Senior answer: measure p95 recovery time from past incidents. Set OPEN wait slightly above that. Serve stale fallback during the window. Argue with data, not opinions.

The Decision Matrix

Circuit Breaker vs Adjacent Patterns

Tuning Knobs That Start Engineering Fights

There is no universal correct row. Document your choice per dependency with measured recovery data from past incidents.

Common Mistakes

1. Setting timeouts too long. A 30-second timeout on a service call that usually takes 20ms means threads block for 30 full seconds before failing. Your thread pool empties in seconds. Set timeouts tight: 2-5 times the p99 response time, not 30 seconds.

2. No fallback when the circuit opens. A circuit breaker that opens and returns a 500 error is better than blocking forever, but it is not great. Define fallbacks so the user experience degrades gracefully rather than failing hard. OPEN contains blast radius. Fallback preserves business function.

3. Retrying without jitter. If 10,000 clients fail simultaneously and all retry after exactly 2 seconds, they hit the recovering service in a synchronized wave. Jitter spreads retries randomly to avoid this thundering herd.

4. Retrying through an OPEN breaker. The breaker already decided the dependency is sick. Retrying wastes caller threads and amplifies load on a recovering downstream. Go straight to fallback when OPEN.

5. Missing slow call thresholds. Circuits that only trip on HTTP 500 errors miss the most dangerous failure mode: a service that responds with 200 but takes 10 seconds. Slow call thresholds catch latency death.

6. Breaker without bulkhead on shared pools. The failures that trip the breaker already consumed threads. By trip time, the pool may already be empty. Deploy both on revenue paths.

7. No circuit breaker monitoring. If you cannot see when circuits are OPEN, you cannot distinguish "the circuit is open protecting us" from "the service is down and we are returning errors." Expose circuit breaker state in health endpoints and dashboards. Alert on unexpected OPEN duration.

🧠 Key Takeaways

• Circuit breaker sits between a service and its downstream dependency. It monitors failures and slow calls. When rates exceed a threshold, it trips to OPEN and returns immediate fallbacks, freeing threads and stopping cascading failure.
• Three states: CLOSED (normal), OPEN (failing fast), HALF-OPEN (testing recovery). Transitions are driven by failure rates, slow call rates, and a wait timeout.
• Timeouts alone are not enough. Threads still block for the full timeout duration on every call until the breaker trips. Breakers return immediately in OPEN state, releasing threads in microseconds.
• Retries pair with breakers carefully. Exponential backoff and jitter on CLOSED state. No retries on OPEN state. Count retry failures toward trip threshold.
• Bulkheads complement breakers. Bulkhead limits pre-trip damage. Breaker limits post-trip damage. One without the other leaves a gap on shared thread pools.
• Resilience4j is the current standard Java library, replacing Netflix Hystrix. Configuration via application.yml. Integrates with Spring Boot Actuator for metrics.
• Graceful degradation means having a fallback hierarchy: cached data, defaults, reduced features, error message, in that order of user impact.
• Chaos engineering validates that your circuit breakers actually work by intentionally injecting failures in staging and production.

Think About It

1. PhonePe's Payment Service calls 4 downstream services: Bank API, Fraud Detection, Notification Service, and Audit Logging. The Bank API has a circuit breaker. When it opens, what should the fallback be? What about Audit Logging: can you serve a degraded payment if Audit Logging is down? What is your compensating action if audit must be durable for RBI compliance?

2. A Swiggy Order Service has a circuit breaker on the Discount Service with a 30-second wait in OPEN state. A Zomato engineer argues the wait should be 5 seconds so users are not locked out of discounts for long. A senior engineer argues 30 seconds is correct. Who is right and why? How would you settle the argument with data from past incidents?

3. You have circuit breakers but no bulkheads. The circuit breaker on the Discount Service opens after 10 failures. But by the time it opens, 500 threads are already blocked from the failures that triggered the trip. What does this tell you about the relationship between breakers and bulkheads? What would you deploy first on a shared Tomcat pool?

Further Reading

• Michael Nygard, Release It!: The book that defined stability patterns including circuit breakers and bulkheads
• Resilience4j documentation: Complete guide to configuring circuit breakers and related patterns in Spring Boot
• Netflix Hystrix Wiki: Historical reference, still useful for understanding the mental model
• AWS re:Invent talks on chaos engineering: Real-world failure injection practices
• Casey Rosenthal and Nora Jones, Chaos Engineering: Comprehensive guide to practicing chaos at scale