Design Distributed Rate Limiter (API Gateway Throttling)

01 /The 09:59:58 AM IRCTC Tatkal bot storm

At 09:59:58 AM every morning, traffic on the IRCTC ticketing platform is normal. At 10:00:00 AM, AC Tatkal booking opens. Within three seconds, traffic spikes from 2,000 requests per second to over 150,000 requests per second.

Over 70% of that incoming traffic is not human travelers clicking train berths. It is automated browser extensions, headless Selenium scripts, and commercial ticket bot networks scraping availability and hammering the /book endpoint.

Without a rate limiter at the gateway:

• Application servers exhaust their database connection pools in four seconds.
• Backend microservices crash with cascading HTTP 504 timeouts.
• Genuine families trying to book an emergency ticket to Varanasi or Patna stare at white error screens.

On my team during a flash sale, we once banned an abusive IP address sending 2,000 requests per second. Within two minutes, customer support got flooded with calls. Hundreds of genuine mobile shoppers across an entire neighborhood in Kanpur were completely locked out. We had banned a shared Carrier-Grade NAT (CGNAT) gateway used by Jio and Airtel mobile towers.

A rate limiter is the defensive perimeter of your infrastructure. It enforces quotas on inbound requests, throttles noisy neighbors, repels brute-force attacks, and returns HTTP 429 Too Many Requests before bad traffic can burn down your downstream databases.

02 /🟢 Layer 1: Core Mechanics & Algorithm Tradeoffs

Client identification in Indian mobile networks

A common beginner mistake is rate limiting purely by client IP address (Remote-Addr).

In India, mobile operators route millions of smartphones through shared Carrier-Grade NAT (CGNAT) pools:

• 10,000 mobile devices in Bangalore can share a single public IPv4 address.
• If you throttle that IP, one malicious bot locks out 9,999 innocent users.

In production, we use a tiered identification strategy:

1. Authenticated endpoints: Key strictly by JWT sub claim or verified user_id (e.g. rl:user:88192:checkout).
2. API partners / B2B merchants: Key by verified api_key or merchant_id.
3. Unauthenticated endpoints: Combine client_ip/24 subnet + TLS Client Hello fingerprint (JA3/JA4) + HTTP User-Agent hash.

Comparing the 5 rate limiting algorithms

Different endpoints require different algorithms:

The Fixed Window boundary attack

If your quota is 100 requests per minute:

• An attacker sends 100 requests at 10:00:59 AM (allowed).
• The window resets at 10:01:00 AM.
• The attacker sends another 100 requests at 10:01:01 AM (allowed).

Result: 200 requests passed within two seconds. If downstream databases were sized for 100 requests per minute, that boundary spike topples them.

Sliding Window Counter calculation

The Sliding Window Counter combines the previous window count with the current window count using fractional elapsed time:

Atomic Token Bucket in Redis Lua

To eliminate race conditions, checking and updating token counts must be atomic. If two gateway pods run GET then SET, they will overwrite each other's decrements.

We execute the logic inside an atomic Redis Lua script:

-- KEYS[1]: Rate limit key, e.g. "rl:user_42:checkout"
-- ARGV[1]: Bucket capacity (e.g. 10)
-- ARGV[2]: Refill rate per millisecond (e.g. 0.01 tokens/ms = 10 tokens/sec)
-- ARGV[3]: Current timestamp in milliseconds
-- ARGV[4]: Cost of this request (usually 1)

local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])

local data = redis.call('HMGET', key, 'tokens', 'last_updated')
local tokens = tonumber(data[1])
local last_updated = tonumber(data[2])

if tokens == nil then
    tokens = capacity
    last_updated = now
else
    local delta = math.max(0, now - last_updated)
    tokens = math.min(capacity, tokens + delta * refill_rate)
    last_updated = now
end

if tokens >= cost then
    tokens = tokens - cost
    redis.call('HMSET', key, 'tokens', tokens, 'last_updated', last_updated)
    local ttl = math.ceil(capacity / refill_rate / 1000) * 2
    redis.call('EXPIRE', key, math.max(60, ttl))
    return { 1, math.floor(tokens) } -- ALLOWED, remaining tokens
else
    redis.call('HMSET', key, 'tokens', tokens, 'last_updated', last_updated)
    return { 0, math.floor(tokens) } -- DENIED, 0 tokens remaining
end

Because Redis executes Lua scripts single-threaded on the key's shard, this check is 100% race-condition free.

03 /🟡 Layer 2: Distributed Architecture & High-Throughput Batching

API headers and contracts

The gateway validates rate limits on every inbound HTTP request and attaches standard RFC response headers:

Successful Request (HTTP 200 OK):

HTTP/1.1 200 OK
Content-Type: application/json
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 74
X-RateLimit-Reset: 1774780860

Throttled Request (HTTP 429 Too Many Requests):

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 14
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1774780860

{
  "error": "RATE_LIMIT_EXCEEDED",
  "message": "Quota exceeded. Please retry after 14 seconds.",
  "retry_after_seconds": 14
}

Sizing the numbers

Interactive rate limiting laboratory

Test all five algorithms against real-time simulated traffic. Switch between steady requests, window-edge bursts, and sustained bot swarms. Watch which algorithms drop requests and how the timeline canvas visualizes allowed versus denied traffic.

High-scale optimization: Local token batching

At 100,000 requests per second, querying Redis on every single request generates 100,000 network round trips. Even with sub-millisecond latencies, Redis network sockets and CPU saturate.

To scale beyond this bottleneck:

1. Local in-memory batching: Gateway pods request tokens in batches of 50 from Redis.
2. The local gateway thread decrements tokens from local RAM in 0.001 ms.
3. When local tokens drop below 10, the gateway asynchronously fetches the next batch of 50 from Redis.
4. Tradeoff: A slight fuzziness in global quota enforcement (an aggressive user might burst up to $50 \times \text{gateway count}$ before being blocked), in exchange for a 50x reduction in Redis network load.

04 /🔴 Layer 3: Architect's Corner

1. Central Redis outage: Fail-Open vs Fail-Closed

If the Redis rate limiter cluster experiences a network partition or hardware failure:

• The API Gateway circuit breaker trips after 3 consecutive socket timeouts (5 ms threshold).
• For critical customer checkout and payment endpoints, the system fails open (allowing traffic through while alerting on-call engineers). It is better to risk higher backend load than to stop the business completely.
• For unauthenticated scrapers or public search endpoints, the system fails closed or degrades to a conservative local in-memory heuristic limit.

2. Clock drift across gateway and Redis nodes

In Sliding Window Counter or Log algorithms, discrepancies between gateway system clocks and Redis server clocks cause inaccurate weight calculations.

• All nodes run the chrony NTP daemon syncing against internal atomic clocks.
• Timestamps used in rate calculations are retrieved directly from the Redis server clock (redis.call('TIME')) inside the Lua script rather than relying on client application clocks.

3. Redis Cluster hash slots and node failover

In a Redis Cluster, keys are sharded across 16,384 hash slots. In plain words: a hash slot is a virtual partition where keys are assigned via CRC16 hash so keys can move between nodes during scaling.

To ensure all keys for a single user land on the same Redis shard (enabling multi-key Lua scripts), we wrap the user identifier in curly braces: rl:{user_88192}:checkout. Redis hashes only the text inside {...} to select the hash slot.

If a Redis primary node crashes:

• Its replica promotes to primary within 1 to 3 seconds.
• During those 2 seconds, the gateway catches the connection timeout and fails open. Rate limiting must never take down the core application.