What an LLM Actually Is

How to read this lesson: New to LLMs? Read only 🟢 first, then Key Takeaways. Already calling models at work? Skim green, sit with yellow and red. English not your first language? Stick to 🟢 and Key Takeaways. Java, .NET, Node, or Python: you are in the right place. This path is built for your stack, not only for Python.

The Refund Policy That Never Existed

A mid sized insurance company in Pune launched a support assistant in October. Simple scope. Customers ask about their policy, the bot answers from the policy documents. The demo went beautifully. Leadership was happy. It shipped in three weeks.

In the second week of live traffic, a customer asked whether he could cancel his health policy and get money back. The bot told him he was entitled to a full refund within 30 days of purchase, no questions asked. Confident answer. Clean formatting. Completely wrong.

The actual policy said 15 days, and only if no claim had been filed. Nobody had written those words about 30 days anywhere. The model produced them because they sounded like the kind of thing an insurance refund clause says.

The customer took a screenshot. He went to the grievance cell with it. The company had to honour it, because a written commitment from their own support channel is hard to argue against in front of an ombudsman. Forty seven customers got the same wrong answer before anyone noticed.

The engineering team's first reaction was the one I hear most often. "The model has a bug, let us report it." There was no bug. The model did exactly what it is built to do. The team had assumed it was a lookup system that fetches facts. It is not.

A large language model predicts the next piece of text, one piece at a time, based on patterns it learned during training. It does not look anything up. When you finish this lesson you will be able to explain why that single sentence changes how you design the whole system around it. You will also know what this whole path is for, what you will be able to build by the end, and why it works if you write Java, C#, or TypeScript every day.

What This Path Is (And Who It Is For)

This is not a course on training models from scratch. It is not a catalogue of clever prompts. It is a path on AI systems architecture: how you put a model behind an API that your company can run in production without inventing refund policies, leaking customer data through a cache, or burning the monthly budget in a week.

What you will learn by the end

Across seven phases you will learn to design and defend:

1. Foundations (this phase): what a model is, tokens, latency, prompts, structured outputs, model choice, and non-determinism.
2. Retrieval and RAG: embeddings, chunking, vector stores, hybrid search, reranking, measurement, multi-tenant isolation.
3. Agents: loops, tools with least privilege, memory cost, multi-agent patterns, human approval, durable execution.
4. Evaluation and safety: eval sets, LLM-as-judge traps, regression testing, prompt injection, DPDP, guardrails.
5. Production operations: traces, latency budgets, caching, rate limits, fallback routing, versioning, incidents.
6. Cost and scale: unit economics under a rupee per request, attribution, batching, cascading, capacity.
7. Case studies and capstone: telco support, document intelligence, ops agents, e-commerce search, and your own architecture pack.

When you finish the path, you should be able to walk into a design review and own the full story: requirements, data flow, retrieval and eval plan, cost model, SLOs, and failure modes. That pack is the proof, not a certificate at the end of a video.

Is this for .NET, Java, and Node developers?

Yes. Explicitly.

Most AI material on the internet is Python first and Python only. A Java engineer who is told to "add RAG" is left translating notebooks. A .NET team is told to learn LangChain before they understand what a retriever is. That is the gap this path fills.

Architecture here is language neutral. Interfaces come first (ModelClient, Retriever, ToolExecutor, and the rest). When a lesson needs real code, it shows the same contract in four ecosystems behind collapsible blocks: Java (Spring AI, LangChain4j), .NET (Microsoft.Extensions.AI, Semantic Kernel), Node and TypeScript (Vercel AI SDK, LangChain.js), and Python (LangChain, LlamaIndex). You can stay in your day job language for the entire path.

If you already ship REST services, queues, and databases, you have the hard skills. You are not late. You need a correct mental model of the model, and the production patterns around it.

How Phase 1 fits

Do not skip ahead to agents or RAG demos until this phase is clear. Phase 1 is the floor. After these seven lessons you should be able to call a model from your stack, explain tokens and latency, treat prompts as versioned contracts, force structured outputs, choose a model under cost and DPDP pressure, and explain why your old unit tests are not enough on their own.

Chai ready? Let us fix the mental model first.

Why Should You Care?

1. Every design mistake in AI systems starts with the wrong mental model. Teams that think an LLM retrieves facts build systems with no retrieval layer, no validation, and no evaluation. Then they act surprised when it invents a refund policy. If you get the mental model right on day one, half the architecture decisions in this path become obvious.
2. This is the first question in any serious AI engineering interview. Not "what is a transformer". The question is closer to "your model gave a confident wrong answer in production, walk me through why and what you would change". A weak answer blames the model. A strong answer explains next token prediction, then talks about grounding, validation and evals.
3. You are being asked to ship these features right now. Almost every backend team in India has an AI item in the roadmap this year. Most of them are being handed a Python notebook and told to productionise it. Understanding what the thing actually does is the difference between shipping a system and shipping a liability.

🟢 The Simple Version (Start Here If You're New)

It Is Your Phone Keyboard, Grown Up

Open WhatsApp on your phone. Start typing "I am reaching in". Your keyboard suggests "10 minutes" or "5 minutes" or "sometime". It has no idea where you are. It has never seen a map. It just knows that after those words, people usually type one of those things.

A large language model is that idea, made enormously better. It was trained on a very large amount of text. From that text it learned which words tend to follow which other words, in which contexts.

In plain words: an LLM (a text prediction engine trained on huge amounts of text) reads everything you send it, then predicts the most likely next chunk of text. Then it adds that chunk to what it has, and predicts again. It repeats until it decides to stop.

That is the whole loop. Read, predict a bit, append, predict again.

Why It Sounds So Confident

Your keyboard suggests one word. The model produces whole paragraphs. Along the way it picks words that fit the tone and shape of what came before.

If the text so far looks like an insurance policy explanation, it will produce text that looks like an insurance policy explanation. Correct phrasing. Sensible structure. Reasonable sounding numbers.

Nothing in that process checks whether the numbers are true. There is no fact database being consulted. The model is not lying, because lying needs intent. It is producing text that fits the pattern.

In plain words: hallucination (a confident answer that is simply not true) is not a malfunction. It is the same machinery that produces correct answers, applied where the model has no grounding.

This is the point most teams miss. Correct output and hallucinated output come out of exactly the same process. The model cannot tell you which one you just got. It has no internal signal that says "I am guessing now".

What It Does Not Have

Three things it does not have, and every one of them will bite you.

It has no memory of your last conversation. Every request starts blank. If your chatbot remembers what the user said two messages ago, it is because your code sent those messages again in the new request. The model did not remember. Your code did.

It has no access to your database, your policy PDFs, or today's date, unless you put that information into the request yourself.

It has no way to tell you how sure it is. A wrong answer arrives with the same tone as a right one.

Training Happened Earlier. Inference Is What You Call.

This is the confusion that creates the worst production designs.

Training is the expensive offline process where a lab feeds huge amounts of text into a model and adjusts billions of internal numbers called weights (the learned parameters that define how the model predicts). That work already happened before you ever opened an API key. You are not training anything when a customer sends a chat message.

Inference is the production call. You send text in, the fixed weights run, text comes out. The weights do not update because a user asked a question. Your support tickets are not "teaching" the model. If you need new behaviour or new knowledge, you change the request (prompt, retrieved context, tools), or you deliberately fine-tune offline, which is a separate project with its own cost and risk. We cover that choice later in this phase and again in the economics phase.

In plain words: the model is a frozen engine at request time. Your job is to feed it the right inputs and check the outputs, not to hope it learns from live traffic.

The keyboard analogy gives you the idea. Now let us look at what a request actually looks like, because that is where the design decisions live.

🟡 Going Deeper

The Request Is The Whole World

Here is the part that reframes everything. A call to a model is stateless. It is one HTTP request. Whatever you put in that request is the entire universe the model can reason about.

POST /v1/chat/completions

system:  You are a support assistant for Acme Health Insurance.
         Answer only from the policy text provided.
user:    Can I cancel and get a refund?

Look at what is missing. The policy text. The customer's plan. The purchase date. The model has none of it. So it fills the gap with whatever pattern fits, and you get thirty days out of thin air.

Now the same call with grounding:

POST /v1/chat/completions

system:  You are a support assistant for Acme Health Insurance.
         Answer only from the policy text below. If the answer is not
         in the text, say you do not know.
context: [Section 7.2] Free look period: 15 days from receipt of
         policy document. Refund available only if no claim has been
         registered. Proportionate risk premium is deducted.
user:    Can I cancel and get a refund?

Same model. Same weights. Completely different reliability, because the facts are now inside the request instead of inside the model's guesswork.

That whole idea has a name, and it is Phase 2 of this path. Retrieval augmented generation (RAG: put the right facts into the request before you ask) is just that discipline with a search system in front of the model call.

Three Different Model Products (Do Not Treat Them As One Blob)

When someone says "call the model," they often mean three different products.

1. Chat or completion models take messages and return generated text (or structured tool calls). This is what people mean by "the LLM" in a support bot.
2. Embedding models take a piece of text and return a list of numbers (a vector) that represents meaning for search. They do not write answers. They power similarity search in RAG. Phase 2 goes deep here.
3. Rerankers take a query plus a short list of candidate passages and score which passages actually answer the query. Cheaper accuracy win after a wide retrieval step.

You pay for them differently, deploy them differently, and evaluate them differently. Using a chat model to "search" by asking it what it remembers is not search. Using an embedding model to write a refund answer is not generation. Keep the product types separate in your head from day one.

Tokens, Not Characters

The model does not read characters or words. It reads tokens (chunks of text, roughly three quarters of a word in English).

"Insurance" might be one token. "Reimbursement" might be three. Hindi and other Indian language text usually costs more tokens per word than English does, because the tokenisers were mostly built with English text in mind. A Hindi sentence can cost two to three times the tokens of the same sentence in English.

This matters for two reasons and both are practical.

Every request has a context window (the maximum number of tokens the model can accept in one call). Send more than that and the call fails, or your oldest content gets dropped silently, depending on your framework.

And you pay per token, both for what you send (input tokens) and what you get back (output tokens). Output is usually priced higher per token than input. That Hindi tax is a real line item if your users type in Hindi.

Here is the packing picture to hold until the next lesson goes deep. Imagine an 8,000 token window. Your system instructions take 1,500. Conversation history takes 2,000. Retrieved policy chunks take 2,500. Tool definitions take 500. You have about 1,500 tokens left for the model's answer. If history grows, something else has to shrink, or the call fails. Lesson 2 works this with real numbers and overflow behaviour.

The Same Input Can Give Different Output

At each step the model produces a probability distribution over possible next tokens. Something like: "15" at 42 percent, "30" at 31 percent, "thirty" at 12 percent, and a long tail.

Then it picks one. How it picks is up to you.

temperature controls how adventurous that pick is. At 0 it takes the highest probability token nearly every time, so the output is close to repeatable. Turn it up and it samples more freely from the distribution, which gives variety and also gives you more chances to wander into a wrong answer.

For a support bot answering policy questions, low temperature. For generating three different marketing subject lines, higher.

Here is the trap. Even at temperature 0 you are not guaranteed identical output across calls. Floating point behaviour on different hardware, batching on the provider side, and silent model updates all cause drift. Providers change the model behind a version label more often than teams expect.

So do not build anything that depends on byte identical output. Your integration tests cannot assert on exact strings. This is the single most common way a team's test suite becomes useless in week three, and it is why Phase 4 exists.

What This Costs You In Latency

A database query returns in single digit milliseconds. A model call is a different category of thing.

The model generates one token at a time, in sequence. Roughly speaking, a 500 token answer takes 500 sequential steps. Typical numbers today are somewhere between 20 and 100 tokens per second depending on the model and load. So a detailed answer takes several seconds, not milliseconds.

You cannot parallelise your way out of that for a single answer, because each token depends on the one before it.

This is why streaming exists. You send tokens to the browser as they are produced, so the user sees words appearing instead of a spinner. The total time is the same. The felt time is much shorter. We cover latency budgets properly in Phase 5.

You now know what one call does. Next, what it means for the systems you are responsible for.

🔴 Architect's Corner

Treat It As An Unreliable Remote Service

Strip away the excitement and describe the model the way you would describe any other dependency in a design review.

It is a remote call over the network. It is stateless. It is slow, in seconds not milliseconds. It is metered per token in both directions. It is non deterministic. It has no correctness SLA. It has rate limits that are aggressive. It can be deprecated by the provider with a few months notice.

Written like that, every senior engineer already knows what to do. Timeouts. Retries with backoff and jitter. Circuit breakers. Fallbacks. Caching. Cost controls. Request tracing. You have shipped all of this before for payment gateways and third party APIs.

The one property with no precedent in your existing toolkit is the missing correctness SLA. Your payment gateway can be slow or down, but when it says a payment succeeded, it succeeded. A model can be fast, healthy, well within rate limits, and confidently wrong. Your monitoring will show green.

That gap is what evaluation and guardrails exist to fill, and it is why they get two full phases in this path rather than a footnote.

The Boundary Question

The most useful architectural question I ask in reviews is simple. What is the blast radius of a wrong answer here?

If a wrong answer means a slightly odd product description, ship it and move on. If a wrong answer means a stated refund commitment, a medical suggestion, or a number that ends up in a compliance report, then the model output cannot be the last step. Something deterministic has to sit after it.

The Pune insurance team eventually shipped a version where the assistant could only answer from retrieved policy sections, and any answer containing a number or a time period had to have that number appear in the retrieved text. If it did not, the response was replaced with a handoff to a human agent.

That check is ordinary code. A regular expression and a substring search. It caught the class of failure that had cost them forty seven bad commitments, and it runs in under a millisecond.

You do not need a smarter model to fix most production failures. You need a boundary.

Silent Model Updates Are A Real Operational Risk

Traditional dependency: you upgrade a library, you choose when, you read the changelog.

Provider hosted model: the thing behind the version label can change without you doing anything. Behaviour shifts. Prompts that worked start producing different formats. Nothing in your deployment pipeline registers a change, because from your side nothing was deployed.

I have seen a team lose two days to this. Their JSON parsing started failing intermittently. No code change on their side, no incident on the provider status page. The model had begun wrapping its JSON in markdown code fences some of the time.

Pin explicit model versions where the provider allows it. Keep a small set of golden test cases that run on a schedule, not only in CI, so drift shows up as a failing check rather than as customer complaints. Phase 5 covers versioning properly.

When Not To Use A Model At All

This gets asked in interviews and gets answered badly.

If the task has a correct answer that a rule can produce, use the rule. Validating a PAN number format is a regular expression. Calculating a premium is arithmetic your actuary already specified. Routing a ticket by keyword is a lookup table. Classifying ten thousand support tickets a day into five fixed categories is a small classifier that costs almost nothing and runs in a millisecond.

Reach for a model when the input is genuinely open ended language, when the output tolerates variation, and when the alternative is a rules engine that nobody can maintain.

The strongest thing you can say in a design review is "we tried this without a model first". Very few teams can say it.

How India-scale systems do it

Public detail on Indian AI architectures is thin, so I will be honest about what is well documented and what is a constraint you should design for.

Language mix is the default, not an edge case. Indian users type in English, in Hindi, in Hinglish, and in regional languages, often inside one sentence. "Mera policy cancel karna hai, refund milega kya?" is a completely normal support message. Any evaluation set built only from clean English queries will pass in testing and fail on real traffic. Build your eval set from actual user messages from day one.

WhatsApp is the interface for a huge share of users. That shapes design more than people expect. No streaming, so the several second generation time is felt fully. Strict message length limits. Rich formatting mostly unavailable. A design that assumes a web chat window with token by token streaming does not transfer.

Data residency is a legal question now. The Digital Personal Data Protection Act, 2023 sets rules on how personal data is handled, and many regulated sectors carry their own localisation requirements. Sending customer records to a model endpoint hosted outside India can be a compliance problem, not just a preference. This decides your model choice before latency or accuracy do. Phase 4 covers it in detail.

Cost per request has to be small. A support interaction that costs 4 rupees in tokens is fine at a thousand a day. At two lakh a day it is 8 lakh rupees a month for one feature, and it will be cut. Indian consumer scale with Indian price sensitivity makes the cost model a first class design constraint rather than a finance problem to sort out later.

Tradeoff: designing for Indian constraints usually pushes you toward smaller models, aggressive caching and tighter output limits. You give up some answer quality on hard questions. What you gain is a feature that survives its first budget review and its first compliance audit.

The Decision Matrix

The row that matters most is the second last one. Model drafts, code verifies. That pattern solves a surprising share of real production problems, and it needs no additional AI at all.

Common Mistakes

1. "The model knows our product." It does not. It knows text patterns from training data that stopped at some date, which may include some public information about your company and almost certainly nothing about your internal policies. Anything specific to you has to travel in the request. Assuming otherwise is how the Pune team got a refund policy that never existed.

2. "We will fix hallucination with a better prompt." Prompting reduces the rate. It does not remove the mechanism. A prompt saying "do not make things up" is a suggestion to a system that has no concept of making things up. Grounding plus validation removes the failure. Prompt wording alone does not.

3. "It worked in the demo." Demos use clean questions from the person who built the feature. Real traffic brings Hinglish, typos, half sentences, and people actively trying to break it. The gap between demo and production is wider for AI features than for any other kind I have shipped.

4. "Temperature 0 makes it deterministic." It makes it much more repeatable. It does not make it deterministic across hardware, provider batching, or silent model updates. Do not write tests that assert exact strings.

5. "We will add evaluation later." Without an eval set you cannot tell whether your prompt change helped or hurt. Teams that skip this end up changing prompts based on the last complaint they received, which moves quality sideways in a random walk. The eval set does not need to be large to be useful. Fifty real questions with known good answers beats nothing by an enormous margin.

6. "The model will learn from our production chats." It will not, not during ordinary inference. Live traffic does not update weights. If quality improves over time, it is because you changed prompts, retrieval, tools, or ran a separate fine-tune job. Hoping the model "picks it up" from support tickets is how teams skip building retrieval and evals.

🧠 Key Takeaways

• This path is AI systems architecture for production, not research and not demo prompts. You will design RAG, agents, evals, ops, and cost, then prove it with an architecture pack.
• Java, .NET, Node, and Python are first class. Interfaces first, then the same contract in your stack. You do not need to become a Python shop to ship this.
• An LLM predicts likely next text. It does not look up facts. Correct answers and hallucinations come from the same machinery, and the model cannot distinguish between them.
• Inference does not train the model. Weights are fixed at request time. New facts go in the request (or through a deliberate fine-tune project), not through live traffic.
• Chat, embedding, and rerank models are different products. Do not treat "the model" as one blob.
• The request is the entire world. No memory, no database access, no knowledge of today. If a fact is not in the request, the model will invent something plausible in its place.
• Treat it as an unreliable remote service. Slow, stateless, metered, non deterministic, and with no correctness SLA. Put a deterministic boundary after it where a wrong answer hurts.

Think About It

1. Your team wants to add an assistant that tells customers their current account balance and recent transactions. Where exactly would you draw the boundary between the model and deterministic code? What would you refuse to let the model generate at all?

2. A product manager asks why the assistant gave two different answers to the same question asked twice. You explain non determinism. She then asks a fair question: "then how do we ever know if it is working?" What do you say, and what would you need to build before you could answer honestly?

3. Your feature costs 3 rupees per conversation and finance has told you to get it under 50 paise. You can change the model, the amount of context you send, the answer length, or add caching. Which do you try first, and what quality are you willing to trade away?

Further Reading

• Andrej Karpathy, Intro to Large Language Models: the clearest explanation of next token prediction for engineers who do not want the maths
• OpenAI: Prompt engineering guide: useful for the mechanics, though treat the reliability claims with the scepticism this lesson argues for
• Anthropic: Building effective agents: a rare vendor post that argues for using less machinery rather than more
• Simon Willison's blog on hallucination and prompt injection: consistently the most honest public writing about what these systems get wrong

Learn it in your stack

• Java: Spring AI reference covers ChatClient and the provider abstractions. LangChain4j is the other mature option.
• .NET: Microsoft.Extensions.AI for the base abstractions, Semantic Kernel when you need orchestration.
• Node and TypeScript: Vercel AI SDK for streaming and provider switching, LangChain.js when you need the wider ecosystem.
• Python: LangChain and LlamaIndex remain the default, with the caveat that both break APIs between versions more often than a backend engineer expects.