The Desi Architect
Notes / Design Interview Problems / Design a Multi-Model AI Gateway (Routing, Quotas, Cost Controls)
Track 5 · Design Interview Problems

Design a Multi-Model AI Gateway (Routing, Quotas, Cost Controls)

A central control plane for enterprise LLM workloads: key pooling, dynamic fallback routing, dual-axis token quotas, semantic caching, and streaming cost attribution.

How to build a production AI gateway that routes millions of LLM requests across model providers, manages enterprise API key pools, enforces RPM and TPM rate limits, slashes token bills with semantic caching, and guarantees zero double-billing during streaming failovers.

Interview problem45 min read
On this page
  1. Why this problem is hard
  2. What are we building?
  3. Entities, API and numbers
  4. Building it up, one step at a time
  5. Where it gets hard
  6. Failure modes at a glance
  7. The whole picture
  8. How interviewers judge this
  9. Questions that come next
  10. Saying it out loud in 45 minutes
  11. Sources and assumptions