The Desi Architect
Notes / Design Interview Problems / Design a ChatGPT-Style Assistant (Streaming, SSE, Conversations)
Track 5 · Design Interview Problems

Design a ChatGPT-Style Assistant (Streaming, SSE, Conversations)

How an AI chat platform streams tokens over HTTP/2 Server-Sent Events, compacts 128k token context windows, routes prompts across model tiers, and protects GPU inference clusters from backpressure.

How one architecture serves 2 crore daily active users, streams tokens at 40 tokens per second across 10 lakh concurrent connections, isolates model tiers, and recovers from network drops without burning extra GPU compute.

Interview problem43 min read
On this page
  1. Why this problem is hard
  2. What are we building?
  3. Entities, API and numbers
  4. Building it up, one step at a time
  5. Where it gets hard
  6. Failure modes at a glance
  7. The whole picture
  8. How interviewers judge this
  9. Questions that come next
  10. Saying it out loud in 45 minutes
  11. Sources and assumptions