Cache Invalidation & Stampede

Ten Thousand Identical Menu Queries at the Same Second

A popular Zomato restaurant page uses Redis with TTL = 3600. At the top of the hour, the key expires. Ten thousand users already have that page open from a push notification. Every request misses cache at once. Every miss queries Postgres for the same menu JSON. Postgres connection pool (max 100) exhausts in seconds. API timeouts cascade. P99 goes from 80ms to 8 seconds.

The restaurant page is down because of a clock.

That is a cache stampede (thundering herd). Invalidation strategy and expiry design are how you avoid serving stale data. Stampede defenses are how you avoid killing the DB when freshness kicks in. Phil Karlton said there are only two hard things in computer science: cache invalidation and naming things. He was not joking.

This lesson covers both sides: when to TTL vs when to DEL, and how to survive the moment 10,000 clients want the same cold key.

Cache invalidation is how you keep cached data fresh. Stampede is what happens when freshness and peak traffic collide. By the end you can pick TTL vs active DEL, and stack jitter, mutex, and warming before the next IPL push.

Why Should You Care?

1. Phil Karlton was right: cache invalidation is genuinely hard. TTL-only without jitter is a time bomb set to peak lunch hour.
2. Stampede is a peak-hour failure mode. Fine at 3 AM, fatal during Swiggy lunch rush, IPL notification burst, or Flipkart sale countdown.
3. Flipkart warms caches before Big Billion Days for a reason. Warming is stampede prevention dressed as an ops runbook. Without it, you test your mutex under live revenue.

🟢 The Simple Version (Start Here If You're New)

IRCTC Tatkal Shape (Different System, Same Herd)

Tatkal opens at 10 AM. Millions click "book" at 10:00:00. The queue absorbs the herd so the DB does not drown. Cache stampede is the same shape without a queue: one expiry event, N identical expensive work items at once.

Your job is either spread the expiry (jitter), serialize rebuild (mutex), serve stale while refreshing (SWR), or preload before the rush (warming).

TTL vs Active Invalidation

In plain words: TTL (Time To Live) means the key expires on a clock. SET key EX 1800. Data may be stale until expiry. Simple. No coupling to every writer service. Good when minutes of staleness are acceptable.

In plain words: active invalidation means writers delete the cache key when source data changes. DEL key on update. Next read rebuilds fresh. Tighter freshness, but every write path must remember to invalidate. Miss one writer and users see stale data until TTL saves you.

The Stampede

In plain words: a cache stampede (also called thundering herd) is when many requests miss the same cold key at once and all hammer the database with the same expensive query.

One cold key + high concurrency = N identical loads. If menu query costs 15ms DB time and 10,000 requests arrive in 200ms, you need 150 seconds of DB time in 0.2 seconds of wall clock. Physics says no.

DB is the shock absorber you did not intend to use. Connection pool maxes. Threads block. Timeouts propagate. Redis is empty and useless at the worst moment.

Defenses

Invalidation keeps data correct. Stampede defenses keep the DB alive when correctness kicks in at peak. Next we walk through the mutex code juniors get wrong and the Flipkart warming runbook.

🟡 Going Deeper

Mutex Pattern (Redis)

async function getMenu(id: string) {
 const key = `menu:${id}`;
 const hit = await redis.get(key);
 if (hit) return JSON.parse(hit);

 const lock = `lock:${key}`;
 const acquired = await redis.set(lock, '1', { NX: true, EX: 15 });
 if (acquired) {
   try {
     const menu = await db.loadMenu(id); // p99 = 12s for mega-chain
     await redis.set(key, JSON.stringify(menu), { EX: 1800 + random(0, 300) });
     return menu;
   } finally {
     await redis.del(lock);
   }
 }
 await sleep(50 + random(0, 50)); // jitter retries
 return getMenu(id);
}

Only one process rebuilds. Others wait and retry. Watch lock TTL vs rebuild time. Lock EX=5s, rebuild takes 12s → at second 6 another worker acquires lock → partial stampede returns. Set EX to p99 rebuild + buffer.

Facebook's memcache paper used leases. Same idea: single flyer rebuilds, others get stale or wait.

Jitter at Scale

Batch import at midnight sets EX 86400 on 50,000 restaurant keys. All expire at next midnight. Stampede at 00:00:00.

Fix on import:

const ttl = 86400 + Math.floor(Math.random() * 3600); // spread over 1 hour
await redis.set(key, value, { EX: ttl });

Cheapest first fix when you discover synchronized TTL. No architecture change. Run it in a migration script tonight.

Stale-While-Revalidate at CDN and App

CDN header:

Cache-Control: max-age=300, stale-while-revalidate=60

CDN serves stale response for up to 60s while fetching fresh copy in background. User never waits on cold origin.

App layer equivalent: if Redis key exists but soft_expired (stored timestamp older than threshold), return stale value to user and fire async refresh. User sees 2-minute-old menu during rebuild. Product call whether that is OK.

Probabilistic Early Refresh

Key has 1800s TTL. In last 180s (10%), each read has 5% chance to trigger background refresh. Load spreads over 3 minutes instead of one second at expiry. Used in large-scale systems (Varnish, some CDN configs). Harder to reason about than jitter but smooths mega-hot keys.

Cache Warming Before Sales

Flipkart at 11 PM before midnight sale: job loads top 50k SKUs into Redis from read replica (not primary). CDN prefetch for hero images. At midnight, 95% hits are warm. Would-be stampede becomes planned, metered load.

Warming fails when hot set prediction is wrong. New viral SKU not in top 50k still stampede. Combine warming with mutex on unknown hot keys.

Invalidation Races (Stale After Update)

Sequence that breaks read-your-writes:

1. Writer updates DB (new price ₹999).
2. Writer DEL cache key.
3. Slow reader still had old value, writes old value back to cache before DB commit visible (or read replica lag).
4. User sees old price ₹1299.

Fixes:

• DEL after DB commit, not before.
• Version in cache key: price:sku:123:v42.
• Short TTL as safety net (60s) even with active invalidation.
• Write-through on price during sale window.

Races and event-driven invalidation are where stale-data bugs hide. The production stacks below are what I have seen survive Big Billion and IPL night.

🔴 Architect's Corner

Event-Driven Invalidation

Price change event on Kafka/SQS → invalidator service deletes keys or updates cache directly. Decouples product writers from Redis knowledge.

At-least-once delivery means duplicate DEL is safe. DEL missing key is fine. Ordering matters for update vs delete: prefer delete after commit event, not before.

Swiggy-style: merchant webhook menu.updated → cache service DEL menu:{id} + optional warm reload for top merchants.

Indian Examples (Production-Shaped)

• Swiggy lunch peak (12:30 to 2:30): jitter on all restaurant keys, mutex on mega-chain menus (McDonald's, Domino's) where rebuild is slow.
• IRCTC tatkal: queue absorbs herd (Phase 3). Same thundering shape, different mitigation layer.
• Hotstar score ticker: sub-second TTL or push via WebSocket. Hourly TTL on live score is stampede bait.
• Flipkart Big Billion: warming runbook at T-2 hours, mutex on deal-of-the-day SKU, purge CDN + DEL Redis on price push event.

War Story: Midnight Expiry Cliff

Food delivery startup (anonymized). Data team bulk-refreshed 80k restaurant keys with identical EX 3600 from a CSV import at 14:00. Every key expired at 15:00. DB CPU 90% for 8 minutes. Auto-scale added read replicas (helped). Proper fix was jitter script + mutex on top 200 chain restaurants. Took 2 days to implement. Incident lasted 8 minutes but repeat was scheduled daily until fix shipped.

When NOT to Mutex

Low concurrency keys: mutex adds latency for no benefit. 3 requests/sec on a cold key does not stampede. Mutex shines at 1000+ concurrent misses on same key.

Also: mutex with Redis during Redis outage fails open or closed depending on your code. Plan fallback (serve stale, circuit break to DB with rate limit).

Warming Job Sketch

// T-2h before sale: run from read replica, not primary
const hotSkus = await db.topSkusByGMV(50_000);
for (const batch of chunk(hotSkus, 500)) {
 const rows = await db.fetchProductBatch(batch);
 await redis.mset(rows.map(r => [`product:${r.id}`, JSON.stringify(r)]));
 await sleep(100); // pace writes to avoid mini-stampede on Redis itself
}

Warm CDN URLs separately (prefetch or origin push). Log keys warmed vs predicted hot set. After sale starts, compare miss rate on unwarmed SKUs.

Double-Checked Locking Variant

Some teams store loading sentinel in cache during rebuild so waiters know to retry shortly instead of hammering DB:

GET miss → SET key="LOADING" NX EX 5 → if ok rebuild else sleep retry

Simpler than separate lock key but mixes data and metadata in same namespace. Document sentinel value in team wiki.

Versioned Cache Keys

On price update to ₹999, bump version in DB. Cache key price:sku:123:v17. Old v16 keys expire naturally. No invalidation race where stale value wins write-back. Cost: more keys in Redis until TTL expires. Good for high-stakes catalog during sales.

TTL vs Invalidate Decision Tree

Does stale data cause money/trust bug?
 Yes → active invalidation or write-through + short TTL safety net
 No → is peak QPS high on same key?
   Yes → jitter + mutex/SWR on that key
   No → TTL alone may be fine

Walk this tree in design review before drawing Redis on the architecture diagram.

Combining Defenses (Production Stack)

Mega-hot restaurant menu during IPL:

1. Jitter on TTL (spread accidental sync expiry).
2. Mutex on miss (one DB rebuild).
3. Local L1 cache 10s on app pod (cut Redis QPS).
4. Warming 30 min before match notification push.
5. Webhook DEL when merchant edits menu (active invalidation for correctness).

No single defense handles all failure modes. Layer them. Document which layer owns which risk in the runbook.

Metrics to Alert On

• Redis miss rate spike on specific key prefix.
• DB QPS on loadMenu query correlating with TTL boundary (top of hour).
• Postgres active_connections near max_connections during lunch peak.
• P99 API latency up while Redis CPU flat (stampede signature: DB is the bottleneck).

Soft TTL vs Hard TTL Pattern

Store { value, cachedAt } in Redis. Hard TTL 3600s (key expires). Soft TTL 300s (app serves stale but triggers async refresh). User never waits on mutex after soft expiry if value still present. Combines SWR with cache-aside in application logic without CDN headers.

const raw = await redis.get(key);
if (!raw) return rebuildWithMutex(key);
const { value, cachedAt } = JSON.parse(raw);
if (Date.now() - cachedAt > SOFT_MS) refreshInBackground(key); // fire-and-forget
return value;

Tradeoff: background refresh can fail silently. Monitor refresh error rate. Stale window = SOFT_MS unless hard TTL expired.

Purge vs Invalidate Vocabulary

CDN purge removes edge copy. Redis DEL removes datacenter copy. Browser cache may still serve until its max-age expires. Full freshness path after price change: DB commit → Redis DEL → CDN purge → optional Cache-Control: no-cache on HTML shell. Missing one layer causes the bug class users report as "price wrong on app but right on website."

The Decision Matrix

Common Mistakes

1. Identical TTL on millions of keys from batch job. Synchronized expiry = synchronized stampede. Run jitter on import.

2. Mutex without lock expiry tuning. Dead lock holder blocks everyone. Or lock expires mid-rebuild and stampede returns.

3. Warming only the CDN, not Redis. App still stampede on API layer. Warm both tiers for hot paths.

4. 1-second TTL "for freshness". Behaves like no cache at peak QPS. DB sees every request. You added Redis latency for nothing.

5. Active invalidation without covering all writers. Admin panel updates DB, mobile API does not DEL. Two write paths, one invalidates. Bug lives in the path that does not.

6. Deleting cache before DB commit. Reader repopulates stale value. Invalidation race. Delete after commit.

🧠 Key Takeaways

• Invalidation: TTL for simplicity, active DEL for tight freshness. Every writer must participate for active mode.
• Stampede = mass miss on same key at once. Deadly at peak. Invisible at 3 AM.
• Jitter is the cheapest spread; mutex, SWR, and warming stack for serious traffic.
• Invalidation races can serve stale after update; version keys or delete after commit.
• Before Big Billion Days or IPL push, warm hot keys or you are testing defenses under live revenue.
• Lock TTL must exceed rebuild p99 or mutex becomes cosmetic.

Think About It

1. 50k keys expire at midnight. Design jitter + warming runbook for Zomato restaurant listings. What do you monitor at 00:05 to know you succeeded?

2. Mutex lock TTL 5s, menu rebuild takes 12s. What happens at second 6? How do you fix without removing mutex entirely?

3. User updates avatar, sees old image for 20 min. TTL-only without invalidation. Diagnose the write path gap and propose fix with tradeoff.

4. Big Billion warming job runs T-30min but sale starts T+0 with new surprise SKU. 40% of midnight traffic hits unwarmed keys. What defenses still apply after warming prediction fails?

At-Least-Once Invalidation Consumers

Queue consumer DEL menu:123 runs twice on duplicate message. Safe. Consumer misses message. Stale until TTL. Pair event-driven invalidation with max TTL safety net (e.g. 15 min on menus) so missed events self-heal. Tradeoff: 15 min stale window on missed invalidation. Monitor consumer lag and dead-letter depth.

Further Reading

• Redis distributed locks doc (mutex caveats, Redlock debate)
• Cloudflare stale-while-revalidate documentation
• Facebook memcache lease paper (classic stampede reference)

Stampede Severity Table

Retries make stampede worse. Clients retry on timeout, doubling effective load. Circuit breaker on DB path during miss storm is ugly but sometimes necessary to protect shared Postgres.

Invalidate-After-Commit Pattern (Transactional Outbox)

BEGIN;
UPDATE menu SET ...;
INSERT INTO outbox(event='menu.updated', id=123);
COMMIT;
-- separate worker reads outbox → DEL menu:123 in Redis

Outbox guarantees invalidation event exists iff DB commit succeeded. Decouples invalidation latency from write path. Tradeoff: eventual cache freshness (worker lag 100 to 500 ms). Acceptable for menus, not for inventory locks during checkout.

Manual Invalidation During Incidents

Ops runs redis-cli DEL 'menu:*' (never KEYS in prod on large cluster; use known key list from warming sheet). Scripted DEL per restaurant ID from incident ticket. Pair with temporary TTL reduction on prefix until root cause fixed. Document who has redis-cli access. Uncontrolled manual DEL during panic can cause stampede if traffic is high. Prefer targeted DEL on known bad keys.

Jitter Migration Script (Copy-Paste Starting Point)

for (const key of keysFromCsvImport) {
 const ttl = 86400 + Math.floor(Math.random() * 7200);
 await redis.expire(key, ttl);
}

Run once after discovering synchronized expiry. Monitor expired_keys rate next midnight. Should spread over 2 hours, not 2 seconds.

Closing Tradeoff (Read This Before the Quiz)

TTL is lazy and simple. Active invalidation is tight and brittle. Stampede defenses add latency and staleness on purpose to save the DB. You cannot maximize freshness, minimize ops, and survive IPL peak on one key without paying somewhere. Pick which pain your product owns: slightly old menu for 60s, or down restaurant page for 5 min.

Re-read caching-patterns write path and this lesson together before Big Billion or IPL push. Invalidation without stampede defense (or the reverse) is half a design.

Load-test stampede explicitly: expire a hot key in staging while 500 virtual users hammer the endpoint. Watch DB connections. Fix in staging, not on the night Swiggy push goes national.

Pair this lesson with distributed-cache-architecture. Mutex on one hot key still hammers one Redis shard if all waiters retry GET together. Stack jitter, warming, SWR, and L1 local cache. Flipkart-scale teams run three or four defenses on the same hot SKU. Build the stack before marketing announces the date.