All About RAG: From Naive Pipelines to Production-Grade Distributed Architecture

Retrieval-Augmented Generation (RAG) is a distributed systems architecture that pairs an information retrieval engine with a Large Language Model (LLM). Instead of trusting an LLM to recall company facts from frozen parametric weights trained months ago, RAG intercepts the user query, fetches verified chunks from external data sources (PDFs, databases, wikis, REST APIs), and injects those exact passages into the model context window before generation.

Think of an unassisted LLM like a brilliant candidate sitting in an interview room with no internet and no company handbook. If you ask them about general banking theory, they speak fluently. If you ask them, "How many casual leaves does Desi Bank grant to a newly joined clerk in 2026?", they either guess generic Indian labor laws or fabricate a confident policy number. RAG hands that candidate the exact page of the official Desi Bank HR Policy PDF right before they answer.

• Overcoming Fixed Knowledge CutoffsPre-trained weights freeze on the day training finishes. RAG injects policies, interest rates, and regulatory circulars updated seconds ago without retraining.
• Eliminating Parametric HallucinationsBy constraining the system prompt to answer strictly from retrieved context, you convert an open-ended text generator into an extractive reasoning engine.
• Accessing Private Enterprise DataInternal HR policies, loan underwriting rules, and customer records never need to be baked into public model weights.
• Verifiable Source CitationsEvery response carries deterministic metadata (source filename, page number, chunk ID) so auditors and customers can verify claims.

At a high level, every RAG execution splits into two runtime stages: the Retrieval Stage (finding the right needles in the haystack) and the Generation Stage (synthesizing a grounded answer from those needles).

Before writing a single line of LangChain or Qdrant code, map out the seven decoupled layers that form a complete RAG pipeline:

Where does RAG deliver immediate ROI in production engineering?

• Internal PDF & Policy Q&AEmployees querying HR leave rules, travel reimbursement caps, and IT security compliance across thousands of internal PDFs.
• Customer Support CopilotsBanking bots answering home loan interest rates, prepayment penalties, and credit card dispute workflows with zero hallucinated fees.
• Enterprise Knowledge SearchUnified semantic search across Confluence, Notion, Jira tickets, Slack threads, and GitHub pull requests.
• Domain-Specific Assistants (Finance, Healthcare, Legal)Underwriters and legal counsel cross-referencing RBI circulars, clinical guidelines, or contract clauses with exact page citations.
• Academic & Technical Research AssistantsSynthesizing multi-paper literature reviews or querying live developer documentation.
• Personalized RecommendationsMatching user intent vectors against dynamic product catalogs and financial instruments.

Foundation models are trained to maximize next-token likelihood across public internet corpora. When deployed directly inside an enterprise without a retrieval layer, they hit six hard architectural walls:

• 1. Fixed Knowledge CutoffThe model has zero awareness of events, RBI rate revisions, or policy circulars published after its training cutoff date.
• 2. Confident HallucinationsLLMs do not have an internal truth-meter; they optimize for plausible-sounding token sequences, fabricating policy clauses when unsure.
• 3. Zero Access to Private Company DataYour proprietary SQL databases, intranet SharePoint folders, and HR PDFs were never in the public training set.
• 4. No Source TraceabilityEven when an unassisted LLM guesses the right answer, it cannot cite a verifiable document file or page number for compliance audits.
• 5. Context Window Cost & "Lost in the Middle"Dumping a 400-page PDF into a 128k context window on every query costs dollars per request and suffers severe attention degradation in the middle tokens.
• 6. Domain-Specific Accuracy GapsInternal acronyms, custom SKU codes, and company-specific formulas are misinterpreted as generic English words.

Compare what happens at runtime when a Desi Bank employee asks: "What is our leave policy?" under Path A (Standalone LLM) versus Path B (Grounded RAG Pipeline):

Every standalone LLM limitation maps directly to a concrete architectural mechanism in the RAG pipeline:

A recurring debate in distributed AI engineering asks: "If models like Gemini 1.5 Pro and Claude 3.5 Sonnet support 1M to 2M token context windows with KV prompt caching, why build a complex RAG pipeline at all?"

In production systems engineering, large context windows and RAG solve fundamentally different problems. Context window is working memory for active reasoning, while RAG is scalable information retrieval with strict boundaries. The decision matrix below compares the structural economics and latency trade-offs between monolithic context dumps and production RAG:

Adding a vector database does not magically make an AI system infallible. In production, Naive RAG pipelines fail in four predictable ways that you must engineer around:

• Garbage In, Garbage Out (Stale/Corrupted Sources)If your S3 bucket contains broken OCR scans, duplicate draft policies, or unpurged 2019 circulars, RAG will faithfully retrieve and cite wrong rules.
• Retrieval Recall FailuresIf the retriever fetches the wrong top-5 chunks (because of bad chunk sizing or vocabulary mismatch), the LLM has zero chance of answering accurately.
• Context Override (Parametric Bias)When a prompt is weakly phrased, the LLM may ignore the retrieved context and revert to its pre-trained internet priors.
• Residual Extractive HallucinationWhen two retrieved chunks discuss different loan products (Home Loan vs Personal Loan), the LLM may accidentally stitch the interest rate of one onto the tenure of the other.

A production RAG system is never a single monolithic script. It is strictly partitioned into two independent workflows that share a common embedding space and vector store:

1. Phase 1 (Offline Asynchronous Ingestion Pipeline): Runs in background workers (Celery, Kafka consumers, or Airflow DAGs). It loads raw documents, cleans noise, splits text into chunks, attaches metadata payloads, computes embeddings in batches, and upserts vectors into the database.

2. Phase 2 (Online Low-Latency Query Pipeline): Runs on the critical user path (FastAPI / Spring Boot). It embeds the incoming user question using the exact same embedding model, runs Approximate Nearest Neighbor (ANN) search, formats the top-k chunks into a grounded prompt, and streams the LLM response.

Walk through the exact millisecond-by-millisecond execution trace when an employee submits a question to the Desi Bank HR Assistant:

• Step 1: User Query ArrivalUser submits: "How many paid leaves do employees get at Desi Bank?" via Web/Mobile client to the API Gateway.
• Step 2: Query EmbeddingThe query string is sent to OpenAI text-embedding-3-small (or local BGE), returning a 1536-dimensional float32 query vector q in ~35ms.
• Step 3: Vector Similarity SearchQdrant compares query vector q against millions of indexed document vectors using Cosine Similarity over its HNSW graph, returning the top-k (k=3) closest IDs.
• Step 4: Context PreparationPayload text and metadata (desi_bank_policy.pdf, Page 5; desi_bank_leave_2026.pdf, Page 2) are extracted, deduplicated, and ordered by relevance score.
• Step 5: Prompt AugmentationThe retrieved passages and the user question are injected into the strict grounding prompt template.
• Step 6: Grounded Answer GenerationGPT-4o (temperature=0) reads the augmented prompt and outputs: "Employees at Desi Bank receive 20 days of paid leave annually (casual, sick, and optional holidays) [Source: desi_bank_policy.pdf, Page 5]."

Your prompt template is the contract between retrieval and generation. Never concatenate raw chunks without explicit role boundaries and refusal rules:

You are an authoritative Desi Bank Enterprise Policy Assistant.
Answer the user's question using ONLY the provided context below.

STRICT RULES:
1. Do NOT use any external knowledge or assumptions outside the Context.
2. If the answer is not explicitly stated in the Context, reply exactly:
   "I don't have enough information in the provided policy documents to answer this."
3. Cite the exact source filename and page number in brackets after every claim, e.g., [desi_bank_policy.pdf, Page 5].

Context:
{retrieved_context_chunks}

User Question:
{user_query}

Grounded Answer:

By separating storage (Vector DB) from reasoning (LLM), your architecture gains modular upgradability: you can swap GPT-4o for Llama-3-70B tomorrow without re-indexing your vectors, or update a single HR PDF in Qdrant in 2 seconds without touching model weights.

Raw enterprise files are messy. PDFs contain repeating headers, page numbers, watermarks, multi-column layouts, and financial tables. If you feed raw uncleaned bytes into an embedding model, your vector space fills with garbage.

Every document must pass through a deterministic 7-stage ingestion pipeline: (1) Load Raw File -> (2) Extract Text/Layout -> (3) Clean Noise (strip headers, footers, HTML tags, extra whitespace) -> (4) Split into Semantic Chunks -> (5) Attach Rich Metadata -> (6) Generate Vector Embeddings -> (7) Atomic Upsert to Vector DB.

A production ingestion service normalizes data from six distinct enterprise categories into a unified Document(page_content, metadata) schema:

How you slice a document determines what the vector database can find. Use the interactive matrix below to compare all 6 chunking strategies across token cost, latency, context preservation, and failure modes:

Experiment with chunk size and overlap on the canonical sentence: "Desi Bank provides savings account, current account, personal loan, home loan and credit card services to its customers." Notice how overlap preserves phrase continuity across chunk boundaries:

Never store naked vectors without metadata. Metadata powers pre-filtering (RBAC security), source citations, and stale-document purging. Below is the canonical Desi Bank metadata payload alongside the runnable PyPDFLoader + RecursiveCharacterTextSplitter implementation:

{
  "source": "desi_bank_policy.pdf",
  "page_number": 5,
  "chunk_id": "chunk_1",
  "title": "Account Types & Leave Policy",
  "department": "HR & Banking",
  "file_type": "pdf"
}

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# 1. Load PDF (creates one Document object per page with page metadata)
loader = PyPDFLoader("desi_bank_policy.pdf")
raw_pages = loader.load()

# 2. Split using hierarchical structural separators
text_splitter = RecursiveCharacterTextSplitter(
    separators=["\n\n", "\n", ". ", " ", ""],
    chunk_size=500,       # Target ~500 characters per chunk
    chunk_overlap=50,     # 10% overlap preserves boundary context
    length_function=len,
    add_start_index=True
)

chunks = text_splitter.split_documents(raw_pages)

# 3. Enrich with enterprise metadata before vectorization
for idx, chunk in enumerate(chunks):
    chunk.metadata.update({
        "chunk_id": f"chunk_{idx + 1}",
        "title": "Account Types",
        "department": "Banking",
        "file_type": "pdf"
    })

print(f"Split {len(raw_pages)} pages into {len(chunks)} enriched chunks.")

A catastrophic limitation of naive text splitters is "Context Amnesia": when a document is chopped into 500-token blocks, individual chunks lose their broader situational context.

Consider an isolated chunk from Desi Bank's policy manual: "The interest rate is 8.5% per annum for repayment tenures up to 30 years with an LTV ratio of 80%." Notice that neither "Desi Bank" nor "Home Loan" appears in this chunk! When an employee queries: "What is the home loan interest rate at Desi Bank?", both dense vector search and BM25 lexical search may miss this chunk because its embedding vector is generic and BM25 matches zero keywords.

Anthropic introduced Contextual Retrieval to solve this: during offline ingestion, an LLM (such as Claude 3.5 Haiku or GPT-4o-mini) inspects the full document alongside the chunk to generate a 50-token contextual prefix situated in the whole file:

• 35% to 49% Fewer Retrieval FailuresAnthropic's empirical benchmarks show Contextual Retrieval cuts the top-20 retrieval failure rate by 35% with dense embeddings alone, and by 49% when paired with BM25 hybrid search.
• Up to 67% Failure Reduction with RerankingCombining Contextual Embeddings + Contextual BM25 + Cross-Encoder Reranking slashes retrieval failures by 67% compared to naive chunking.
• Ingestion Cost OptimizationLeverage prompt caching on the full document context during ingestion batching so 100 consecutive chunk calls share the same cached document prompt, cutting LLM ingestion costs by up to 90%. The math: an 80,000-token policy PDF split into 200 chunks means 200 calls × 80,000 tokens = 16 million input tokens for one document if you resend the whole PDF every time. Keep the document as a fixed prefix, change only the chunk at the tail, and cached reads are billed at a fraction of normal input price (the cache write costs a little extra once). Savings of 85% to 90% are typical, but check your provider price sheet. Illustrative, not a quote.

from openai import OpenAI

client = OpenAI()

def generate_contextual_chunk(full_document_text: str, chunk_text: str) -> str:
    """Uses a fast model to prepend 50-token situational context to an isolated chunk."""
    prompt = f"""<document>
{full_document_text[:4000]}
</document>
Here is the chunk we want to situate within the whole document:
<chunk>
{chunk_text}
</chunk>
Please give a short, succinct context (under 50 tokens) to situate this chunk for search retrieval."""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[{"role": "user", "content": prompt}]
    )
    context_prefix = response.choices[0].message.content.strip()

    # Prepend context before embedding and BM25 indexing
    contextualized_chunk = f"{context_prefix}\n\n{chunk_text}"
    return contextualized_chunk

# Raw chunk without context:
# "The interest rate is 8.5% per annum for tenures up to 30 years."

# Contextualized chunk ready for Qdrant and BM25:
# "In Desi Bank's 2026 Retail Lending Policy, under Home Loan interest rates and terms:
# The interest rate is 8.5% per annum for tenures up to 30 years."

In real projects, three document structures routinely break naive text splitters:

• Financial Tables Split Mid-RowIf a home loan interest rate table is split at character 500, row 8 loses its column headers. Fix: Serialize tables into Markdown or HTML <table> blocks and keep them atomic.
• Multi-Column PDF Reading OrderStandard PDF extractors often read straight across left and right columns on the same line, scrambling sentences. Fix: Use layout-aware parsers (Unstructured hi_res or PyMuPDF blocks).
• Context Fragmentation ("The Pronoun Problem")Chunk 1 says "Desi Bank Home Loan Policy." Chunk 4 says "It carries an 8.5% interest rate." Because Chunk 4 only says "It", it fails retrieval for "Home loan rate". Fix: Prepend document/section titles to every chunk.

Computers cannot measure the semantic similarity between the strings "Desi Bank home loan interest rate and eligibility documents" and "housing mortgage paperwork criteria" using character matching, because they share zero words. An Embedding Model solves this by mapping text into a dense array of floating-point numbers (e.g., 384, 768, 1536, or 3072 dimensions) where concepts with similar meaning sit at nearby coordinates.

Every string passes through five deterministic stages inside the embedding encoder: (1) Text Normalization/Preprocessing -> (2) Subword Tokenization (Byte-Pair Encoding / WordPiece into token IDs) -> (3) Transformer Neural Forward Pass (contextual token vectors) -> (4) Mean or [CLS] Pooling across token positions -> (5) L2 Unit Vector Normalization (scaling length to ||v|| = 1).

Choosing an embedding model locks in your vector database schema; changing models later requires re-embedding and re-indexing your entire document corpus. Compare the 8 industry-standard embedding models below:

When high-dimensional Desi Bank vectors (1536d) are projected onto 2D space (via UMAP/t-SNE), documents naturally self-organize into semantic neighborhoods: Banking Documents, HR Documents, Finance Policies, and IT Documents. Click or filter clusters below to see how a query vector lands inside its matching neighborhood:

How does a vector database decide which chunk vector B is closest to your query vector A? By computing a geometric distance metric across all d dimensions:

from openai import OpenAI
import numpy as np

client = OpenAI()

text = "Desi Bank home loan interest rate and eligibility documents"
response = client.embeddings.create(
    model="text-embedding-3-small",
    input=text
)

vector = response.data[0].embedding
print(f"Dimensions: {len(vector)}")          # Output: 1536
print(f"First 4 values: {vector[:4]}")       # Output: [0.0182, -0.0341, 0.0561, 0.0084]
print(f"L2 Norm: {np.linalg.norm(vector):.4f}") # Output: 1.0000 (Unit normalized)

When a user query vector arrives, a naive Brute-Force (Flat) search computes the cosine similarity against every single vector in the database. At 10,000 vectors, brute force takes 3 milliseconds. At 50 million vectors (1536d), brute force scans 300 GB of float32 arrays per query, taking 4+ seconds and melting your CPUs.

Vector Databases solve this using Approximate Nearest Neighbor (ANN) indexes, most notably HNSW (Hierarchical Navigable Small World), which organizes vectors into a multi-layer skip-list proximity graph, finding the Top-k nearest neighbors in O(log N) hops.

Do not pick a vector database based on Twitter hype. Evaluate Qdrant, pgvector, Pinecone, Milvus, Weaviate, FAISS, Chroma, and MongoDB Atlas against your vector count, metadata filtering complexity, and operational budget:

In interviews and capacity planning meetings, you will be asked: "How much RAM do we need to index 10 million chunks?" Use the formula and interactive calculator below to compute exact memory requirements for float32, Scalar Quantization (int8), and Binary Quantization:

Compare Cosine, Dot Product, and Euclidean (L2) metrics alongside Flat, IVFFlat, HNSW, and IVF_PQ indexing structures:

At enterprise scale (10M to 100M chunks), storing raw uncompressed 1536-dimensional float32 vectors in RAM becomes cost-prohibitive. Vector Quantization compresses floating-point arrays into compact representations with minimal loss in retrieval recall.

The breakdown below compares the three primary quantization strategies used in high-scale vector engines like Qdrant and Milvus:

• The Secret to Binary Quantization: Two-Stage Oversampling & RescoringBinary Quantization alone loses subtle magnitude differences. Production engines solve this via Two-Stage Search: Stage 1 searches the 1-bit binary index in RAM to pull the Top-50 candidates in <2ms, then Stage 2 rescores only those 50 candidates using the original float32 vectors streamed from NVMe SSD (memmap) to return the final Top-5 with 99.5% precision.
• When to Use Scalar int8Default choice for 95% of production systems between 1M and 50M vectors: slashes node RAM costs by 75% with zero perceivable loss in business retrieval accuracy.
• Binary Quantization Is Not a Drop-In FlagBQ works on high-dimensional embeddings (1024 dims and above, such as OpenAI 1536 or 3072, or Cohere v3). On small models (384 or 768 dims) recall drops a lot. To recover recall, oversample: ask for limit × 3 or 4 binary candidates (for a final Top-20, fetch about 80), rescore them with the original float32 vectors kept memory-mapped on NVMe, then return the Top-20. In Qdrant this is rescore=True with oversampling=3.0. Only the binary codes live in RAM. What you give up: an NVMe read per rescored candidate, so p99 now depends on disk, not just RAM.
• When to Use Binary QuantizationIdeal for 50M+ vectors on models explicitly trained with high directional dispersion (such as Cohere embed-v3 or OpenAI text-embedding-3-large), fitting 50 million chunks into a single 32GB RAM commodity node.

from qdrant_client import QdrantClient
from qdrant_client.models import (
    VectorParams, Distance,
    ScalarQuantization, ScalarQuantizationConfig, ScalarType,
    BinaryQuantization, BinaryQuantizationConfig, HnswConfigDiff
)

client = QdrantClient(url="http://127.0.0.1:6333")

# Option A: 4x RAM Reduction with int8 Scalar Quantization (99%+ Recall)
client.create_collection(
    collection_name="desi_bank_scalar_quantized",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
    quantization_config=ScalarQuantization(
        scalar=ScalarQuantizationConfig(
            type=ScalarType.INT8,
            quantile=0.99,       # Clips extreme 1% outlier tails for higher resolution
            always_ram=True      # Keep int8 vectors in RAM for ultra-fast ANN search
        )
    ),
    hnsw_config=HnswConfigDiff(on_disk=True)  # Store original float32 vectors on SSD
)

# Option B: 32x RAM Reduction with Binary Quantization + Two-Stage Oversampling
client.create_collection(
    collection_name="desi_bank_binary_quantized",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
    quantization_config=BinaryQuantization(
        binary=BinaryQuantizationConfig(
            always_ram=True      # 1-bit vectors live in RAM (192 bytes per chunk!)
        )
    )
)

Below is the exact JSON storage layout inside Qdrant (ID + Vector + Payload) and the complete Python implementation for creating a collection, upserting Desi Bank vectors with metadata payloads, and running filtered Top-5 similarity search:

{
  "id": 1,
  "vector": [0.12, -0.34, 0.56, 0.08, "... (1536 float32 values)"],
  "payload": {
    "text": "Employees at Desi Bank are entitled to 20 days of paid leave per calendar year as per the Desi Bank HR policy.",
    "source": "desi_bank_policy.pdf",
    "page_number": 5,
    "chunk_id": "chunk_1",
    "title": "Account Types & Leave Policy",
    "department": "HR & Banking",
    "file_type": "pdf"
  }
}

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, MatchValue

# 1. Connect to containerized Qdrant cluster (never use :memory: in production)
client = QdrantClient(url="http://127.0.0.1:6333")

COLLECTION_NAME = "desi_bank_policies"

# 2. Create Collection with 1536d Cosine HNSW index
if not client.collection_exists(COLLECTION_NAME):
    client.create_collection(
        collection_name=COLLECTION_NAME,
        vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
    )

# 3. Upsert vectors with rich metadata payload
client.upsert(
    collection_name=COLLECTION_NAME,
    points=[
        PointStruct(
            id=1,
            vector=query_or_doc_embedding,  # 1536-dim float list
            payload={
                "text": "Employees at Desi Bank get 20 days paid leave (casual, sick, optional).",
                "source": "desi_bank_policy.pdf",
                "page_number": 5,
                "chunk_id": "chunk_1",
                "title": "Account Types & Leave Policy",
                "department": "HR & Banking",
                "file_type": "pdf"
            }
        )
    ]
)

# 4. Query Top-5 nearest neighbors with RBAC/Department pre-filter
results = client.search(
    collection_name=COLLECTION_NAME,
    query_vector=user_query_embedding,
    query_filter=Filter(
        must=[FieldCondition(key="department", match=MatchValue(value="HR"))]
    ),
    limit=5
)

for hit in results:
    print(f"Score: {hit.score:.4f} | Source: {hit.payload['source']} (p.{hit.payload['page']})")
    print(f"Content: {hit.payload['text']}\n")

Relying solely on Dense Vector Search is one of the most common mistakes in early RAG systems. Dense embeddings compress an entire 500-word chunk into a single smooth semantic vector. That makes them brilliant at matching synonyms ("car" <-> "automobile", "home loan" <-> "housing mortgage"), but terrible at exact character matching.

If a Desi Bank auditor searches for circular "RBI/2024-25/DOR.CRE.REC.104" or policy code "HR-LV-09", a dense embedding model treats that alphanumeric string as rare out-of-vocabulary subword tokens and retrieves generic banking pages. Sparse Lexical Search (BM25) uses an inverted index to match exact tokens with surgical precision. Production RAG combines both into Hybrid Search.

Best Matching 25 (Okapi BM25) is the probabilistic ranking formula powering Elasticsearch, OpenSearch, Lucene, and Qdrant sparse vectors. Unlike naive TF-IDF, BM25 introduces term-frequency saturation (so repeating "leave" 50 times does not give 50x score) and document-length normalization:

Compare Dense Search, BM25, Hybrid Search, Multi-Query Expansion, Step-Back Prompting, HyDE, MMR, and Cross-Encoder Reranking across latency, token overhead, recall, and precision:

Below is the clean Python retrieval implementation connecting OpenAI query embeddings to Qdrant search, setting the stage for Multi-Query, HyDE, MMR, and Cross-Encoder Reranking in Module 10:

from openai import OpenAI
from qdrant_client import QdrantClient

openai_client = OpenAI()
qdrant = QdrantClient(url="http://127.0.0.1:6333")

def retrieve_policy_chunks(query: str, top_k: int = 5, score_threshold: float = 0.72):
    # 1. Convert user query to 1536d vector
    emb_res = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=query
    )
    query_vector = emb_res.data[0].embedding

    # 2. Retrieve Top-k chunks above minimum similarity threshold
    hits = qdrant.search(
        collection_name="desi_bank_policies",
        query_vector=query_vector,
        limit=top_k,
        score_threshold=score_threshold
    )

    return [
        {
            "text": hit.payload["text"],
            "source": hit.payload["source"],
            "page": hit.payload["page"],
            "score": round(hit.score, 4)
        }
        for hit in hits
    ]

In generic ChatGPT usage, prompt engineering is about unlocking creativity and brainstorming. In enterprise RAG, creativity is a bug. Your prompt is a defensive firewall designed to force the LLM to act as a closed-book extractive synthesizer.

A production RAG prompt must enforce four non-negotiable invariants: (1) Context Exclusivity (ignore pre-trained internet memory), (2) Deterministic Refusal (explicitly state when the retrieved context lacks the answer), (3) Verifiable Attribution (attach [source, page] to every claim), and (4) Prompt Injection Immunity (treat retrieved text as untrusted data, never as system instructions).

Switch between the 4 prompt paradigms below: (1) Simple Naive Prompt, (2) Role + Strict Guardrail Prompt, (3) Bullet-Point Structured Prompt, and (4) Few-Shot Grounded Prompt, to see how prompt structure transforms LLM output reliability:

You are an authoritative Desi Bank HR Policy Officer.
Answer the question using ONLY the provided context.
If the answer is not explicitly in the context, reply:
"I cannot find this in the official Desi Bank policy documents."
Always cite [source, page] after every statement.

Context:
{context}

Question: {question}

Combine these 8 techniques to harden your generation layer against edge cases:

Always set temperature=0 (greedy decoding) in RAG pipelines so identical contexts yield deterministic answers:

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

# 1. Define strict system + human prompt template with XML context delimiters
rag_prompt = ChatPromptTemplate.from_messages([
    ("system", """You are an authoritative Desi Bank Policy Assistant.
Use ONLY the following <context> passages to answer the user's question.
If the answer cannot be found in the <context>, reply exactly:
"I cannot find this information in the official Desi Bank policy documents."
Always cite the source filename and page number in brackets, e.g., [desi_bank_policy.pdf, Page 5].

<context>
{context}
</context>"""),
    ("human", "{question}")
])

# 2. Initialize deterministic LLM (temperature=0 is mandatory for factual RAG)
llm = ChatOpenAI(model="gpt-4o", temperature=0)

# 3. Compose LCEL chain
chain = rag_prompt | llm
response = chain.invoke({
    "context": "[desi_bank_policy.pdf, Page 5]: Employees receive 20 days paid leave annually (casual, sick, optional).",
    "question": "What is the leave policy at Desi Bank?"
})
print(response.content)

Now we connect Modules 1 through 8 into a single working software blueprint for Desi Bank. The system takes raw PDF/DOC/TXT policy files, processes and chunks them, embeds them with OpenAI, indexes them in Qdrant, retrieves top-k relevant passages per query, and generates a cited answer via GPT-4o.

Use the interactive code tabs below to switch between Tab 1 (Reference RetrievalQA implementation matching standard interviews) and Tab 2 (Modern Production LCEL with RunnableParallel and explicit source citation formatting):

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_qdrant import QdrantVectorStore
from langchain.chains import RetrievalQA
from langchain_core.prompts import PromptTemplate

# Step 1: Load Desi Bank PDF Policy Documents
loader = PyPDFLoader("desi_bank_policy.pdf")
documents = loader.load()

# Step 2: Split into 500-char chunks with 50-char overlap
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(documents)

# Step 3 & 4: Generate OpenAI Embeddings & Index into Qdrant Vector DB
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = QdrantVectorStore.from_documents(
    documents=chunks,
    embedding=embeddings,
    url="http://127.0.0.1:6333",
    collection_name="desi_bank_policies"
)

# Step 5: Configure Top-3 Similarity Retriever
retriever = vector_store.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 3}
)

# Step 6: Build Grounded RetrievalQA Chain with GPT-4o (temperature=0)
prompt_template = """Answer ONLY using the context below. Cite source and page number.
If unknown, say "I don't know based on the provided policy."

Context: {context}
Question: {question}
Answer:"""

PROMPT = PromptTemplate(template=prompt_template, input_variables=["context", "question"])
llm = ChatOpenAI(model="gpt-4o", temperature=0)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever,
    return_source_documents=True,
    chain_type_kwargs={"prompt": PROMPT}
)

# Execute Verification Query
result = qa_chain.invoke({"query": "What is the leave policy at Desi Bank?"})
print("Grounded Answer:", result["result"])
for idx, doc in enumerate(result["source_documents"], 1):
    print(f"{idx}. {doc.metadata['source']} (Page {doc.metadata.get('page', 1)})")

When executed against the Desi Bank policy corpus, the pipeline returns grounded answers with exact file and page citations, and safely refuses out-of-domain questions:

Users rarely type well-formed search queries. An employee might type a 3-word fragment ("leave policy?"), a misspelled query, or a multi-part question. Because a single vector embedding of a 3-word query is sparse in semantic signal, basic Top-k similarity search often misses relevant policy sections that use formal terminology ("Annual Paid Time Off Entitlement").

Query Expansion & Multi-Query Retrieval use a fast, lightweight LLM (e.g., gpt-4o-mini) to rewrite the user query into 3 to 5 rich semantic variations ("What is leave policy, paid leave, vacation, casual leave, sick leave, holiday policy"), executes vector searches for all variations in parallel, and merges the deduplicated results.

Two powerful query transformations bridge the gap between short user questions and detailed policy documents:

• Step-Back PromptingWhen a user asks a narrow, specific question ("Can I carry forward 12 unused casual leaves from 2025 to 2026?"), Step-Back Prompting first generates a broader principle query ("What are the Desi Bank leave policy rules and carry-forward limits?") so the retriever pulls the foundational policy table alongside specific clauses.
• Hypothetical Document Embeddings (HyDE)Questions and answers have different grammatical structures (Query-to-Document asymmetry). HyDE asks an LLM to generate a hypothetical ideal policy paragraph answering the question (even if details are guessed), embeds that hypothetical document, and searches Qdrant using Document-to-Document vector similarity, landing right inside the true policy cluster!

from langchain_openai import ChatOpenAI
from langchain.retrievers.multi_query import MultiQueryRetriever

# 1. Expand "What is leave policy?" into multiple rich variations
llm_fast = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# Direct expansion example:
# Input:  "What is leave policy?"
# Output: "What is leave policy, paid leave, vacation, casual leave, sick leave, holiday policy"

# 2. Attach strict Metadata Pre-Filter on the base Qdrant retriever
filtered_retriever = vector_store.as_retriever(
    search_type="mmr",  # Maximal Marginal Relevance prevents near-duplicate chunks
    search_kwargs={
        "k": 5,
        "fetch_k": 20,
        "lambda_mult": 0.7,  # 0.7 relevance vs 0.3 diversity balance
        "filter": {
            "department": "HR",
            "doc_type": "policy"
        }
    }
)

# 3. Wrap in MultiQueryRetriever for parallel query fan-out + union deduplication
multi_retriever = MultiQueryRetriever.from_llm(
    retriever=filtered_retriever,
    llm=llm_fast
)

Why do production RAG pipelines retrieve Top-25 chunks first and then shrink them to Top-5 using a Reranker? Because of the fundamental architectural difference between a Bi-Encoder (Embedding Model) and a Cross-Encoder (Reranker):

A Bi-Encoder embeds the Query and the Document separately into two vectors and computes a fast dot product, great for scanning 10M chunks in 15ms, but it loses token-by-token cross-attention between the query words and document words. A Cross-Encoder (like BAAI/bge-reranker-large or Cohere Rerank v3) feeds [CLS] Query [SEP] Document [SEP] together into a Transformer so every query token attends to every document token. Running a Cross-Encoder on 10M chunks would take hours, so we use Two-Stage Retrieval: Stage 1 (Hybrid Vector + BM25) fetches the top 25 candidates in 20ms, and Stage 2 (Cross-Encoder) re-scores those 25 pairs in 80ms to pick the golden Top-5.

from sentence_transformers import CrossEncoder

# Initialize Cross-Encoder Reranker model
reranker = CrossEncoder("BAAI/bge-reranker-large", max_length=512)

def two_stage_retrieve_and_rerank(query: str, initial_candidates: list[dict], top_n: int = 5):
    # Stage 1 has already retrieved top-20 candidates via Hybrid (Vector + BM25)
    # Stage 2: Score every (query, chunk_text) pair jointly with full self-attention
    pairs = [[query, doc["text"]] for doc in initial_candidates]
    scores = reranker.predict(pairs)

    for doc, score in zip(initial_candidates, scores):
        doc["rerank_score"] = float(score)

    # Sort descending by Cross-Encoder score and return Top-N
    reranked = sorted(initial_candidates, key=lambda x: x["rerank_score"], reverse=True)
    return reranked[:top_n]

In production enterprise RAG, two stubborn pain points break traditional Bi-Encoder pipelines: (1) loss of fine-grained token-level matching when condensing 500 words into a single 1536d vector, and (2) OCR failure on complex multi-column layouts, financial balance sheets, and architectural flowcharts.

Two modern retrieval paradigms solve these failure modes by replacing single-vector embeddings with multi-vector Late Interaction:

• ColBERT (Contextualized Late Interaction over BERT)Indexes documents as bags of 128-dimensional token vectors. Queries execute fast matrix dot products in vector stores supporting multi-vectors (such as Qdrant multivector search) to match exact policy phrases without losing semantic context.
• ColPali: Vision-Driven Document Retrieval Without OCRTraditional pipelines pass PDFs through Tesseract or PyMuPDF, destroying visual hierarchy and splitting table rows. ColPali (built on PaliGemma-3B) feeds high-resolution page images directly to a vision encoder. It creates patch embeddings capturing text, table lines, bold headers, and diagram geometries simultaneously.
• Bypassing Brittle PDF ParsingColPali queries like "What is the home loan eligibility table in Desi Bank policy?" match the visual patch tokens directly against the PDF page, retrieving complex tables with 100% fidelity where standard OCR produces garbled noise.
• Production Latency Trade-OffColPali and ColBERT require storing ~1,000 multi-vectors per page. Always pair Late Interaction with Binary Quantization or use ColPali as a Stage-2 visual reranker over candidate PDF pages.

Let me tell you what happens a month after you ship hybrid search with a reranker. An officer in Pune asks: "Can a probationary officer claim reimbursement for a broken laptop screen?" The system answers "Yes, up to Rs 15,000 within 7 working days." HR calls you the next morning. The answer was wrong.

Dense search found the right chunk. BM25 agreed. The reranker put it at number one. So what broke? The sentence "Applicable only to confirmed Grade-B officers and above" sat in the previous chunk. Your text splitter cut the page between the rule and its exclusion. The chunk that won the ranking never carried the condition that mattered.