Why RAG Exists: Fine Tune Versus Retrieve

The Fine Tune That Knew Last Year's Premiums

A mid sized general insurer in Gurgaon wanted a WhatsApp bot that could answer policy questions in Hindi and English. The product manager had watched three conference talks about fine tuning. The plan sounded clean. Collect a few thousand support tickets. Fine tune a mid sized model. Ship. No extra infrastructure. "The model will learn our policies," someone said in the design review, and half the room nodded.

They spent eight weeks cleaning tickets, redacting PII, and running training jobs on a rented GPU box. Demo day looked sharp. The bot answered free look questions in the company's tone. It used the right product names. Legal smiled at the phrasing.

Two months later IRDAI circulars forced a change to the free look window on one product line. Ops updated the PDF on the intranet the same afternoon. The bot kept quoting the old window for eleven days. Nobody had retrained. Nobody had a pipeline to retrain. Finance had already called the first fine tune "done." A second training cycle meant another data scrub, another eval pass, another sign off from compliance. While that conversation crawled, agents were still pasting the correct clause from SharePoint and the bot was still inventing confidence around the old number.

The second failure was quieter. A customer asked about a rider that launched after the training cut off. The model had never seen that rider. It did not say "I do not know." It produced a paragraph that sounded like their other riders, with a made up waiting period. Support discovered it only when the customer filed a complaint citing the bot.

Fine tuning taught the bot how to talk like the company. It did not give the bot a live shelf of facts. Retrieval (fetching the right text at question time and putting it in the request) is the pattern that keeps facts current. That pattern, put together as RAG (Retrieval Augmented Generation: retrieve relevant text, then ask the model to answer using it), is why this whole phase exists.

When you finish this lesson you will know what fine tuning actually buys you, what retrieval actually buys you, why fine tuning to "add knowledge" is usually the wrong spend, and what you give up when you choose retrieval instead.

Why Should You Care?

1. Interviews ask this decision almost every time RAG comes up. "Would you fine tune or use RAG?" is not a brand preference question. A strong answer separates behaviour from facts, then talks cost, freshness, and latency.
2. Real teams burn quarters on the wrong lever. I have seen more than one India product team fine tune because a blog said so, then discover their rate cards change monthly and their training set is already a museum.
3. Production pain hides in the update path. Wrong answers from a stale fine tune look like model bugs. They are usually process bugs. If you cannot say how a fact update reaches the user, you do not have a knowledge system.

🟢 The Simple Version (Start Here If You're New)

Two Different Jobs, One Confused Budget

Think of a new junior who joins your support floor in Pune.

Fine tuning is the induction training. You teach how we greet customers, how we structure a reply, which words legal hates, how we handle angry WhatsApp threads. After induction, the junior sounds like us. That is behaviour and format.

Retrieval is the filing cabinet next to the desk. Rate cards, policy PDFs, IRDAI notes, product FAQs. When a customer asks about a premium, the junior opens the right file and reads. That is facts.

Nobody smart tries to make the junior memorise every premium table so you can throw away the filing cabinet. Premiums change. Products launch. Circulars land on Friday evening. You update the file. You do not re-run induction every time a number changes.

In plain words: fine tuning changes how the model behaves by training it further on your examples. Retrieval supplies the text the model should use for this question, at request time, without changing the model weights.

In plain words: RAG means you search your documents for passages that match the question, paste the best ones into the prompt, and ask the model to answer only from that material (plus clear rules when the material is missing).

What Fine Tuning Is Good At

Fine tuning shines when the hard part is style, structure, or a narrow skill, not a moving library of facts.

• Always answer in a fixed JSON shape your backend already parses.
• Match a brand voice that prompting alone cannot hold for long answers.
• Classify tickets into your messy internal taxonomy.
• Extract fields from claim forms the same way every time.
• Speak in a mix of Hindi and English the way your agents actually write, without five paragraphs of prompt instructions every call.

Notice what is common. The "right answer" is a way of working, not "today's premium for plan X in pin code 560001."

What Retrieval Is Good At

Retrieval shines when the hard part is knowledge that lives outside the base model, or knowledge that changes faster than you want to retrain.

• Policy wordings and endorsements
• Product catalogues and rate cards
• Internal runbooks and SOPs
• Ticket history for this customer (with isolation rules)
• Release notes, error code lists, partner circulars

In plain words: if a human agent would open a document or a tool to answer, you probably want retrieval or a tool call, not a fine tune.

The Mistake That Started The Gurgaon Story

The team treated "know our policies" as a training problem. Policies are a data problem with an update clock. Training freezes a snapshot into weights. Snapshots go stale. Stale weights still sound confident. That combination is expensive and dangerous in regulated domains.

A Picture You Can Keep

Customer question
       |
       v
[ Retrieve top passages from your docs ]  <-- facts, updated when docs update
       |
       v
[ Build prompt: rules + passages + question ]
       |
       v
[ Model generates answer ]  <-- behaviour from base model + prompt (+ optional fine tune)

Fine tuning sits inside the last box. It changes the generator. It does not replace the retrieve step for living knowledge.

The simple split is clear. Next we put real numbers on cost, latency, and freshness, because that is where architecture reviews get honest.

🟡 Going Deeper: Behaviour Versus Facts

Fine Tune Path, End To End

A serious fine tune is not "upload a CSV and pray."

1. Collect examples that show the behaviour you want (inputs and ideal outputs).
2. Clean PII. Under DPDP (India's Digital Personal Data Protection Act), training data is still personal data if it contains customer content.
3. Split train and holdout. If you only measure on the training set, you will celebrate overfitting.
4. Run training. Pay for GPU time or a vendor fine tune API.
5. Eval on tasks that matter: tone, schema validity, refusal behaviour, Hindi or Hinglish quality.
6. Pin a model ID. Ship behind a flag. Watch online metrics.
7. When behaviour needs to change, collect new examples and do it again.

That loop is weeks for a careful team, not an afternoon. It is the right loop for stable skills. It is a terrible loop for weekly fact updates.

Retrieval Path, End To End

A serious RAG path looks more like search infrastructure.

1. Ingest documents (PDF, HTML, Confluence, CMS export).
2. Chunk them into passages (next lessons cover this in depth).
3. Embed passages into vectors (again, next lessons).
4. Store vectors and metadata in a vector index or database.
5. At question time, embed the question, retrieve top k passages, optionally rerank.
6. Build a prompt that includes passages and hard rules: cite, refuse if missing, do not invent clauses.
7. Generate. Log which chunks were used. Measure whether the right chunks even arrived.

When IRDAI changes a clause, you update the source document and re-ingest. No GPU training job. Minutes to hours if your pipeline is healthy, not another quarter.

Cost Comparison With Rough India Numbers

These are order of magnitude numbers for a design review, not a vendor quote.

I keep seeing finance teams compare only "training invoice versus no training invoice." They miss the ongoing cost of retrain cadence, and they miss the per request token tax of stuffing retrieved text into every call. Both sides have a bill. The bills land in different months.

Freshness Is An SLO, Not A Feeling

Write the update budget in the design doc.

• Rate card changes: must reflect within 24 hours of CMS publish.
• Compliance circulars: must reflect within 4 hours of legal sign off.
• Bug fix in a runbook: same day.

If your only knowledge path is fine tune, those SLOs force a training factory you will not staff. If your path is retrieval, those SLOs become pipeline and ownership questions: who publishes, who re-indexes, what happens when parse fails.

Latency Budget

RAG adds steps before generation:

• Query embedding: often tens of milliseconds hosted, more if self hosted under load
• Vector search: single digit to low tens of milliseconds for modest corpora, more with heavy filters
• Optional rerank: can add 50 to 200+ ms
• Longer prompts: generation may slow because the model reads more tokens

For a WhatsApp support bot, users tolerate a couple of seconds. For autocomplete inside a claim form, they do not. Fine tuning that removes retrieval can win on latency for narrow tasks where the model truly does not need external text. That is a real tradeoff, not a slogan.

Hybrid Is Normal

You will not pick pure fine tune or pure RAG for every feature.

Common production shape I defend in reviews:

• Base or lightly instruction tuned model for general ability
• Optional small fine tune or strong prompting for output schema and tone
• RAG (or tools) for any fact that can change or must be cited
• Deterministic code for money math, eligibility rules, and anything that must not freestyle

Fine tune and retrieve are not enemies. Confusing their jobs is the enemy.

A Tiny Sketch Of The Decision In Code

// Pseudocode: facts come from retrieve, not from hope
async function answerPolicyQuestion(q: string, tenantId: string) {
  const passages = await retriever.search({
    query: q,
    filter: { tenantId, docType: "policy" },
    k: 6,
  });

  if (passages.length === 0) {
    return { text: "I do not have this in the policy pack. Please talk to an agent." };
  }

  return modelClient.complete({
    modelId: "support-bot-v3", // pinned; maybe fine tuned for tone
    messages: [
      {
        role: "system",
        content:
          "Answer only using PASSAGES. If missing, say you do not know. Quote clause ids.",
      },
      { role: "user", content: formatPassages(passages) + "\n\nQuestion: " + q },
    ],
  });
}

The fine tune, if any, lives in support-bot-v3. The truth lives in retriever.search.

You now have mechanics and cost shape. The architect level failure modes are where teams still get burned after they "do RAG."

🔴 Architect's Corner

Fine Tuning To Stuff Knowledge Is A Smell

I am not anti fine tune. I am anti using fine tune as a content management system.

Signals you are stuffing knowledge into weights:

• Your training set is mostly copied policy paragraphs with light rewrites
• Stakeholders ask "when will the model learn the new circular" the way they ask about a CMS publish
• You have no chunk level citations in the product UI
• Legal wants a paper trail of which document version produced an answer, and you only have "the model said"

Weights are a terrible CMS. No useful versioning per clause. No partial update. No easy audit of "which sentence supported this refund claim."

When I Still Push Fine Tune First

• You need a strict output contract and prompt alone drifts after 2k tokens of context
• You are compressing a huge instruction manual of house style into the model so every request stays cheap
• You run a classification or extraction task at huge volume where a small specialised model beats a large general model on cost
• You have a stable task and a team that will own the retrain loop with evals

Even then I still keep retrieval for living reference data unless the task truly needs none.

The Hidden Cost Of "We Will Retrain Monthly"

Monthly retrain sounds adult in a slide deck. In practice:

• Someone must refresh and re-label data
• PII scrubbing regresses
• Eval sets rot
• Offline metrics look fine while online ticket deflection drops
• You pin nothing and a base model upgrade shifts behaviour under the same fine tune name

If the business change rate is weekly, monthly retrain means you are wrong most of the time. Say that out loud in the review.

Retrieval Is Not Free Of Ops

RAG fails in ways fine tune people underestimate.

• Bad chunking splits a table so no passage has the full premium row
• Embeddings miss exact policy numbers (hybrid search lesson later)
• Stale index after a publish
• Wrong tenant's chunk sneaks past a filter (multi tenant lesson later)
• Prompt ignores retrieved text and still hallucinates (you need evals and refusals)

Choosing RAG means you chose to operate search. Budget an owner. "The LLM team" is not an owner if they only write prompts.

War Story: The Rate Card Weekend

A lending services company I advised had fine tuned on last quarter's fee schedule to "make answers accurate." Sales ran a weekend campaign with a promotional processing fee. Marketing updated the website. The model kept quoting the old fee. By Monday morning, chat and WhatsApp had promised the old number to a few hundred leads. Ops had to honour the cheaper fee or fight on social media. The retrain was still "scheduled for next sprint."

We ripped facts out of the fine tune scope the same week. Fine tune kept tone and JSON. Fees moved to a retrieved rate card document with a published_at timestamp and a hard rule: if rate card age is older than 24 hours against CMS, refuse and page on call. Ugly, but honest. Incident rate collapsed.

What Marketing Decks Skip

• Fine tune demos love fixed FAQs. Production knowledge is not a fixed FAQ.
• "Grounded on your data" in a vendor slide might mean "we fine tuned once on a zip file you uploaded in March."
• Lower training prices do not fix the human process around data and eval.
• RAG demos love a single PDF. Production is twenty systems, partial permissions, and ugly scans.

Security And Compliance Angle

Fine tune datasets often contain real tickets. That is a second copy of customer pain, sitting in object storage, copied into vendor training endpoints, sometimes retained longer than your ticket system. Retrieval keeps the system of record in places you already govern, if you design permissions correctly. Neither path is automatically compliant. Retrieval is usually easier to align with existing document ACLs and retention, which matters under DPDP and sector rules.

How To Argue This In A Design Review

Bring three slides worth of substance, not a brand preference:

1. What changes, and how often. If the answer is weekly or faster, fine tune as CMS is already losing.
2. What must be cited. If legal or grievance cells need a document id, retrieval (or tools) is mandatory.
3. Unit cost and latency budget. Show rough paise per request with and without retrieved passages, and the p95 you can afford on WhatsApp.

If someone still wants to "just fine tune the PDFs," ask them to write the retrain SLO next to the business change SLO. The silence after that question is usually the decision.

The tradeoff for this lesson, stated without romance: retrieval keeps facts current and citeable, and you pay with added latency, more prompt tokens, and a search stack to run. Fine tuning can make behaviour reliable and sometimes remove a retrieve hop, and you pay with training cost, freeze-in-time knowledge, and a slow update path.

How India-scale systems do it

UPI and bank support. Intent and "how do I complete KYC" style behaviour can be prompted or lightly specialised. Balance, mandate status, and dispute state come from tools and core systems, not from a fine tune. Policy explainers for products retrieve the latest approved wording. Residency and audit matter more than leaderboard IQ.

IRCTC and travel. Fare rules and cancellation slabs change. A bot that memorised last festival's slabs is a complaint generator. Retrieve the rule pack for that train type and quota. Spikes at tatkal time also punish heavyweight multi step RAG if you do not cache safe universal answers carefully.

Insurance (the opening story). Circular driven wording is a retrieval problem with legal ownership of the source PDF. Tone and bilingual phrasing can be fine tuned or carefully prompted. Claim status is a tool call into the core system, never a paragraph the model "recalls."

Swiggy, Zomato, and large commerce support. Menu prices and restaurant status change too fast for retrain loops. Order state is a tool. Help centre articles are RAG. Classification of "where is my order" versus "refund for cold food" is a small model or classifier task, sometimes fine tuned for cost at volume.

Hotstar or OTT help during big matches. The same five account questions dominate. Cheap models plus retrieval over a small, hot FAQ corpus beat a giant fine tuned model that still needs updates when the login flow changes.

Enterprise SaaS sold to Indian SMEs. Each tenant has different policy PDFs. Fine tuning one model on all tenants is how you leak phrasing across customers in subtle ways. Per tenant retrieval with hard filters is the default shape. Fine tune, if any, stays on shared behaviour only.

Tradeoff at India scale: change velocity on products, regulations, and catalogues is high, language is messy, and compliance wants an audit trail. Retrieval fits that world. You give up some pure generation latency and accept search quality as a product metric.

The Decision Matrix

Every row hides the same core tradeoff: currency and control of facts versus simplicity and sometimes latency of a self contained model call.

Common Mistakes

1. "We fine tuned on our PDFs, so we have RAG." You have a frozen snapshot in weights. That is not retrieval at question time, and it will not pick up tomorrow's PDF without another training cycle.

2. "RAG means we never need to fine tune." If schema compliance and tone are failing under load, a small behaviour specialised model can still be worth it. RAG does not fix a model that ignores instructions half the time.

3. "Bigger fine tune dataset always means better answers." More duplicated policy text often means more memorisation of old wording and more PII risk, not better judgement. Quality and freshness beat bulk.

4. "If the model sounds sure, the fact is current." Confidence is a style of text. It is not a freshness signal. Only your index timestamp, tool response, or document version can speak to currency.

5. "We will fix wrong facts in the system prompt." A giant prompt becomes a second unversioned CMS. It is hard to eval, easy to overflow, and still does not scale per tenant. Put facts in documents and retrieve them.

🧠 Key Takeaways

• Fine tuning teaches behaviour and format. It is induction training for the model, not a live library of premiums and clauses.
• Retrieval supplies facts at question time. Update the document and re-index when the world changes.
• RAG is retrieve, then generate with that context. The model still can ignore or mangle passages, so you need rules, citations, and evals.
• Fine tuning to add knowledge is usually the wrong spend when your corpus changes faster than your retrain cadence.
• RAG costs latency, tokens, and search ops. You trade that for freshness, citations, and a saner compliance story.
• Hybrid is normal: specialised behaviour in the model, living facts in retrieval or tools.

Think About It

1. Your fintech updates merchant fee schedules every Monday morning from a central CMS. Product wants a "smart" fee explainer bot by next month. What is your default architecture, and what SLO do you write for fee freshness?

2. A colleague argues that fine tuning on the last six months of tickets will make retrieval unnecessary, because "the model will have seen every issue." What fails in that plan for a new product launch next week, and what fails for DPDP?

3. You have a small fine tune that produces perfect JSON for claim intake, and a RAG path that sometimes returns the wrong policy PDF section. Leadership asks which project to fund next. How do you decide without turning it into a pure model quality debate?

Further Reading

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: the original RAG paper, useful for vocabulary even if your stack looks nothing like 2020 research code
• OpenAI: Fine tuning guide: practical view of what vendors think fine tuning is for (and what they warn against)
• Anthropic and other lab notes on reducing hallucinations with grounding: read for the operational idea of "put the evidence in the prompt," not for one vendor lock in
• Your own incident postmortems on wrong policy answers: the best internal reading if you have them

Learn it in your stack

• Java: Spring AI and LangChain4j both expose chat models plus document ingest and retriever abstractions. Start with a ChatClient or ChatLanguageModel for generation, and a content retriever for passages. Keep fine tuning as a separate model deployment id you pin in config.
• .NET: Microsoft.Extensions.AI gives you IChatClient for generation. Semantic Kernel adds memory and text search patterns that map cleanly to RAG. Fine tuned endpoints are just another client registration with a different deployment name.
• Node and TypeScript: Vercel AI SDK for model calls, plus LangChain.js when you want ingest and retriever chains. Store your "behaviour model id" and "embedding model id" as config, not string literals scattered in routes.
• Python: LangChain and LlamaIndex have the densest RAG tutorials. Use them to learn the pipeline, then make sure ownership of the index and the update SLO still sits with your team, not with a notebook.