Sahayak Capstone: Build Milestones

The Helper You Can Actually Break

Most AI courses end with a slide deck. You design a box diagram, name a vector store, and never prove that a forgotten tenant filter can leak a competitor's price list. Sahayak (Hindi for helper) is meant to be the opposite: a multi-tenant support assistant you build against frozen interfaces, with milestone tests that fail when isolation, tools, human gates, budgets, or trajectory honesty are missing.

Under plan/sahayak you get a learner starter that fails milestones on purpose, a reference solution that passes, and fixtures (40+ chunks, 80+ labelled queries, 30+ trajectories). There is still no live LLM required for the gates. Treat this lesson as the map of what you must implement, how to run red then green, and how the architecture pack should talk about your code.

By the end you will name milestones 1–8, run starter (expect red) and reference (expect green), and connect the architecture pack to tests rather than adjectives.

Why Should You Care?

1. Interviews ask for proof, not only boxes. "Show me the test that stops cross-tenant retrieve" beats naming a vendor.
2. Production AI is interfaces plus gates. Retriever, ToolExecutor, limits, HITL hash, trajectory eval: that is the job.
3. A pack without gates is theatre. "We isolate tenants" is empty if milestones 2 and 3 cannot pass on your code.

🟢 The Simple Version (Start Here If You're New)

What Sahayak Is Trying To Be

In plain words: Sahayak is a learning capstone for multi-tenant support RAG. Many companies share one service. Each company's documents and tools stay isolated. Money tools need approval. Agents cannot claim they raised a refund without calling the tool.

In plain words: a milestone is an automated check that a property still holds after your code changes. Not a demo script. Not a Confluence hope.

What Is In The Repo Today

plan/sahayak/
 README.md
 package.json
 src/stable-interfaces.ts      # frozen contracts
 src/starter/compose.ts        # YOUR work (npm test imports this)
 src/reference/compose.ts      # staff solution (self-check only)
 fixtures/tenants.json         # 42 chunks, 2 tenants + markers
 fixtures/eval/labelled-queries.json
 fixtures/eval/golden-transcripts.json
 fixtures/policies/pgvector-schema.sql
 tests/milestones.test.ts      # milestones 1–8

Run:

cd plan/sahayak
npm test                 # starter — expect FAIL until you implement
npm run test:reference   # reference — expect PASS

Honest Limits (Still)

• No live model calls for milestones. You are testing gates, not chat UX.
• In-memory retrieve with keyword + soft rank fusion. Optional SQL sketch for pgvector is in fixtures. Wire real ANN later if you want portfolio depth.
• Reference is for unblocking yourself, not for submitting as "I built it." Interviewers can ask you to explain your starter diffs.

The Seven Milestones (Map)

In plain words: if any milestone fails, you do not call the system production-ready for money or multi-tenant data.

Milestone 1: Interfaces Compose

stable-interfaces.ts freezes Retriever, ToolExecutor, ToolContext, and friends. Your implementation should type-check against those shapes. When Spring AI or LangChain renames helpers, your facade stays.

Failure mode without it: every feature imports a different vendor type. Swaps become rewrites.

Milestone 2: Refuse Empty Tenant

if (!query.tenantId || !query.tenantId.trim()) {
 throw new Error('REFUSE_RETRIEVE: tenantId is required');
}

No default tenant. No "search all for debug" in the same method production uses.

Failure mode without it: one forgotten call site searches the shared pool. That is the multi-tenant-rag incident pattern.

Milestone 3: Isolation With Markers

Fixtures plant ACME_INTERNAL_ONLY and BETA_INTERNAL_ONLY. Query as A for B's marker. Expect zero hits. Results should stamp metadata.tenantId.

Failure mode without it: you only unit-test that a WHERE string exists in source code. Side paths still leak.

Milestone 4: Authz Deny

listAllowedTools and execute both respect ctx.roles. createRefund is not available to orders:read alone. The model can propose anything. The executor decides.

Failure mode without it: prompt-only "do not refund" lines fail under jailbreak or confused-deputy pressure.

Milestone 5: Payload Hash

hashAction(toolName, args) // canonical JSON of sorted keys
// approval stored with that hash
// execute recomputes hash; mismatch => fail closed

Approval of amount 500 does not authorize amount 999.

Failure mode without it: Slack "LGTM" while the tool payload changed in a retry.

Milestone 6: Tool Budgets

checkToolBudget enforces maxToolCallsTotal and maxCallsPerTool. A getOrder storm stops at the cap.

Failure mode without it: weekend cost spikes and dependency meltdown, as in the agent-failure-modes lesson.

Milestone 7: Trajectory Honesty

If the assistant text claims a refund was raised or filed, and createRefund never ran, score hard-fails with fake_side_effect.

Failure mode without it: final-answer evals stay green while users get fiction.

Walkthrough: Break Then Fix Isolation (Hands-On)

Do this once so the markers stop being abstract.

1. Open fixtures/tenants.json. Note the ACME and BETA secret phrases.
2. Run npm test. Confirm green.
3. Temporarily change MemoryTenantRetriever.retrieve to ignore tenantId and search all tenants (union of maps). Save.
4. Run npm test again. Milestone 3 should fail. Read the assertion message.
5. Revert. Confirm green.

That loop is the entire point of a capstone kit: you feel the red, then you restore the property. When starter mode lands, you will start from red by design.

Walkthrough: Hash Mismatch On Refund

1. In the test or a small scratch script, register approval for { amountInr: 500, orderId: 'O1' }.
2. Execute createRefund with amountInr: 999 and the same order id.
3. Expect PAYLOAD_HASH_MISMATCH (or equivalent ok:false error string).
4. Execute with the original args. Expect ok.

Money paths without this pattern are how "we approved a small refund" becomes a large one after a model retry rewrites args.

Learner Track (What You Do)

1. Run npm test on starter. Confirm red (usually M2 first).
2. Implement gates in src/starter/compose.ts until npm test is green.
3. Use npm run test:reference only as a hint, not as a submit button.
4. Optional: implement SQL retriever from pgvector-schema.sql.
5. Write the architecture pack about your starter, with tradeoffs.

Rough time: 6–12 h for milestones, 6–12 h for pack, more if you wire a real store.

How This Connects To Earlier Lessons

Sahayak is not a new theory dump. It is the place those lessons stop being optional.

🔴 Architect's Corner

Green Reference Is Not Your Capstone

I have seen teams demo a WhatsApp bubble while a debug endpoint still searched all tenants. I have also seen "capstone repos" that only prove the author can run someone else's green tests. Sahayak is set up so starter fails and your implementation passes. Shipping a screenshot of test:reference is not the portfolio story.

Why Thin Reference Still Has Value

Even a skeleton forces vocabulary: tenant refuse, payload hash, trajectory honesty. Use it in design reviews this week. Do not pretend it is a production bot.

Portfolio That Survives Scrutiny

A strong close for the path:

1. Architecture pack (next lesson) with numbered tradeoffs.
2. Fork of Sahayak where your code passes milestones.
3. One measured eval report (pass rate by tag, even on synthetic fixtures).
4. Explicit list of what you did not build (no live model, no full corpus) so you do not oversell.

Hours Messaging

Reading the path plus quizzes and pack writing is roughly 30–40 hours. Implementing the Sahayak starter through green milestones and a solid pack is typically another 20–40 hours, depending on whether you stop at in-memory gates or wire pgvector and more tools.

Tradeoff: in-memory gates teach isolation and authz fast. Real ANN and model wiring teach ops and cost. Do both only if you have the hours.

How India-scale systems do it

B2B SaaS support looks like Sahayak: shared infra, strict tenant retrieve, money tools gated.

UPI and collections care need trajectory evals that catch "I raised the refund" lies.

Festival peaks need budget milestones so one tenant cannot stampede tools.

BFSI security questionnaires ask logical vs physical isolation. Your milestone 3 answer should point at tests, not adjectives.

Tradeoff: more milestones slow the first green bar. Fewer milestones ship demos that fail the first enterprise review.

The Decision Matrix

Common Mistakes

1. "I will add tenantId later." Milestone 2 exists because later never comes cleanly.

2. "Approval is a Slack emoji." Milestone 5 needs a hash. Emoji is theatre.

3. "Final answer eval is enough." Milestone 7 catches side-effect fiction.

4. "Green npm test means I finished the capstone." If you only ran the reference, you validated the staff solution, not your build.

5. "I rewrote interfaces to match my framework." Then you left the path contracts. Adapt frameworks to interfaces.

Minimum Isolation And Money Checklist

Before you call any fork "production-shaped":

1. Empty tenant refused on every retrieve path.
2. Cross-tenant marker test on BM25-like and vector-like paths if you add both.
3. Write tools denied without role.
4. Write tools denied without matching payload hash approval when gated.
5. Per-tool and total budgets enforced in the loop.
6. Trajectory hard-fail on claimed side effects without tools.
7. No superuser pool for app traffic if you add Postgres RLS.

Tradeoff: this checklist costs a few days. Skipping it can cost a trust incident.

A Suggested Implementation Order (So You Do Not Drown)

Do not try to invent a full product on day one. Order the work so each red bar teaches one property.

Day 1 to 2: Red bar, then tenant refuse. Run npm test. Read the first failure (usually empty tenantId). Implement refuse-empty. Re-run. Celebrate one green line, not the whole suite.

Day 2 to 3: Isolation. Keep the ACME and BETA markers sacred. Implement tenant-scoped retrieve and metadata stamps. Run until milestone 3 passes. Then deliberately break isolation once so you never forget the feel of red.

Day 3 to 4: Tools and roles. listAllowedTools and execute must agree. Deny createRefund for orders:read. Do not "fix" this by widening roles in the test context.

Day 4 to 5: Hash and budget. Canonicalise args, hash, register approval, re-check on execute. Then wire checkToolBudget so six getOrder calls stop at five.

Day 5 to 6: Trajectory honesty and fixtures. Implement scoreTrajectory hard fails. Confirm milestone 8 still sees isolation probes and lie trajectories on the expanded fixture set.

Day 6 to 8: Pack. Open the architecture pack lesson. Fill six sections using your starter and the measured fixture counts from your last green run. List what you skipped (live model, real ANN).

If you burn more than about twelve focused hours on gates alone, stop adding features and finish the pack. Depth beats a half-wired chat UI.

How To Talk About Sahayak In An Interview

Bad answer: "I cloned the repo and tests were green."

Better answer: "I implemented tenant refuse, isolation markers, ToolExecutor deny, payload hash, budgets, and trajectory hard fails in the starter until npm test passed. Reference stayed a self-check. Fixtures were N chunks and M labelled cases. I did not claim live model quality."

Bring one diff story: the first time hash mismatch blocked a refund amount change. That is more senior than naming LangGraph.

Common Portfolio Anti-Patterns

Screenshot of chat only. Without milestone proof, chat is theatre.

Only test:reference green. That proves the staff solution, not your work.

Deleted isolation markers to "make tests simpler." You deleted the exam.

God-mode tools in starter for demos. Interviewers will ask who is the principal.

Silent rewrites of stable-interfaces.ts. Then your code is not on the path contract.

Mapping Hours To Outcomes

That is why the path can defend about 20–40 hours for a serious Sahayak plus pack once the kit is real. If you only read the lesson and run reference, claim hours honestly as reading time, not build time.

🧠 Key Takeaways

• Sahayak is the path's hands-on isolation and agent-gate lab, under plan/sahayak.
• Milestones 1–8 cover compose, tenant refuse, isolation, authz, HITL hash, budgets, trajectory honesty, and fixture probes.
• Starter fails; reference passes. Your job is the starter.
• Interfaces stay frozen in stable-interfaces.ts. Implementations may start thin.
• Architecture packs should describe systems that can pass these tests, not only draw them.
• Tradeoff: in-memory gates vs full retrieval stack. Name what you built and what you skipped.

From Green Gates To A Real Product (Optional Stretch)

When milestones are green, you have a gate laboratory, not a full support product. Stretch only if you still have hours:

1. SQL retrieve. Use fixtures/policies/pgvector-schema.sql, load chunks with embeddings, keep the same Retriever interface and re-run isolation tests.
2. ModelClient stub. One pinned model, structured output, redacted logs. Do not remove hard gates because the model "sounds careful."
3. Eval harness exit codes. Wire labelled queries into the regression harness from earlier lessons (exit 0/1/2).
4. Desk vs customer catalogs. Two allowlists on the same tool server idea (MCP lesson). Prove customer sessions cannot see desk tools.

Stop stretch work if the pack is still empty. A measured pack plus green starter beats a half-wired chat.

Think About It

1. You add a semantic cache. Which milestones must still pass, and what new test do you add for cache keys?

2. Your company uses Java in production. How do you keep milestone meaning while reimplementing compose?

3. Milestone 7 passes but users still get wrong policy text. Which layer is untested, and how would you extend fixtures?

Further Reading

• plan/sahayak/README.md and tests/milestones.test.ts
• Lessons: multi-tenant-rag, designing-agent-tools, human-in-the-loop, agent-failure-modes, evaluating-agents
• Capstone architecture pack (next in phase 7): write the pack as if Sahayak (or your fork) were the system under review
• Authoring policy quiz bias rules: every new quiz must pass the length-bias script

Learn it in your stack

• Java: reimplement Retriever and ToolExecutor as Spring beans. Keep the same milestone table in JUnit.
• .NET: same contracts as interfaces, xUnit for the seven asserts.
• Node and TypeScript: start from plan/sahayak. Prefer extending starter when it exists; until then, fork compose carefully.
• Python: port compose to protocols. pytest for the seven asserts. Do not let LangChain globals skip tenant context.