The Project: Hollin Answers
Every lesson in this course is set at one company, on one project. Hollin Home Appliances is made up. All people, products, documents, cases, prices, dates and numbers in the project are fictional and simulated. The real cases that open each lesson are separate, cited, and clearly marked as real.
The company
Hollin Home Appliances Ltd makes small kitchen and home appliances in the north of England: two blenders, a smart kettle, an air purifier and a coffee machine. It does not run a shop. Its appliances are sold by retailers; Hollin looks after everything that happens after the sale: warranty claims, repairs, spare parts, safety notices and product questions.
Repairs are done at Hollin's own repair centre and by independent service partners, local repair firms that Hollin pays for warranty work. Two of them appear in this course: Brook Street Repairs in Leeds and Calder Fix in Halifax.
Money in this course is in pounds sterling. Dates in the text are written day first (9 October 2026); dates in data are written year first (2026-10-09).
The problem
Hollin's knowledge is spread across nine places: the public help centre, product manuals (several only as PDFs), the service partner portal, a ticket system, a community forum, a parts price export, a finance file share, partner email, and the repair case database. Support agents search them by hand. They get it wrong in ways that matter:
- On 12 September 2026 an agent told a customer the X300 Pro motor was covered for 2 years. That came from the 2024 warranty terms, archived but still on the help centre. The current cover is 5 years (case 30501).
- On 14 September 2026 Hollin recalled X200 jars from batch B-2291. Agents answering questions about cracked jars did not always find the notice.
- Repair partners phone the support line for service bulletins they cannot easily find.
In August, Kwame built a quick prototype: every document embedded, the top three by similarity, straight to a language model. Six agents tried it from 21 September to 2 October 2026. Half of the ten logged queries in 06-shared-data.md ended with the agent escalating. It served stale terms, leaked finance-only cost prices to agents, put an injected forum post first, and answered a question the documents could not answer.
Grace, Head of Customer Support, wants it built properly. You are the engineer who builds it.
What Hollin Answers must do
- Answer questions from Hollin's documents, and only from them, with a citation for every claim.
- Say "I don't know" and hand over to a person when the documents do not answer the question.
- Never show anyone a document they are not allowed to see: customers see public documents, partners see their own, finance-only files stay in Finance.
- Prefer the current version of a document over an archived one, and never serve a withdrawn one.
- Treat document text as data, never as instructions.
- Look up live facts (a repair's status) in the case database, not in the documents.
- Prove each of the above with measurements, and keep proving it as the documents change.
The people
| Name | Role | In the project |
|---|---|---|
| Grace Okoye | Head of Customer Support | The owner. Brings the problems her agents and customers have. Signs off what ships. |
| Kwame Asante | Knowledge and data engineer | Owns the source systems and their exports. Built the prototype. Works with you on ingestion and indexing. |
| Hannah Price | Product Safety and Quality Manager | Owns safety notices and service bulletins. Cares most about stale and withdrawn content. |
| Farah Iqbal | Security and Compliance Lead | Asks who can see what, what an attacker could plant, and what is logged. |
| Owen Fletcher | Finance Manager | Asks what each answer costs. Owns the finance-only cost prices. |
| Lucy Hart | Technician, Brook Street Repairs (partner) | Asks repair questions. Must never see Calder Fix's documents. |
| Dan Mercer | Owner, Calder Fix (partner) | Wrote the pump claim email. The other side of the tenant boundary. |
| Ellie Shaw | Customer | Owns an X200 that keeps showing E7. The course's running customer. |
Minor characters (a support agent, a forum poster, another customer in the case table) may appear in a single lesson. The main cast above does not change.
Hollin's sources
| Source | Holds | Format | Who may see it | Changes |
|---|---|---|---|---|
| Help centre | Policies, help answers, notices, the price list | Web pages (HTML) | Everyone | Weekly |
| Manuals | Specifications, error codes, user guides | Text, Markdown and PDF | Everyone | When a product changes |
| Service portal | Service bulletins | Text | Partners and staff | Monthly |
| Partner portal | Partner handbook, partner agreements, partner email | Text and email | Each partner sees its own; staff see all | Occasionally |
| Ticket system | Support case notes | Text | Staff only | Every hour |
| Finance share | Internal cost prices | Text | Finance only | Quarterly |
| Community forum | Customer posts | Web pages | Everyone; written by anyone | Every day |
| Parts shop | Spare parts and retail prices | CSV export | Everyone | Monthly |
| Case database | Repair cases and their status | SQL table | Staff; customers see their own | Every minute; never embedded |
Products
| Code | Product | Model number | Things to remember |
|---|---|---|---|
| X200 | X200 Blender | HL-X200-UK | Error codes E4, E7, E9; jar recall for batch B-2291 |
| X300 | X300 Pro Blender | HL-X300-UK | Error codes E4 (a different meaning) and E12; pump bulletin SB-112 |
| K50 | K50 Smart Kettle | HL-K50-UK | Pairs with the Hollin Home app; bulletin SB-114, withdrawn 8 October 2026 |
| P9 | P9 Air Purifier | HL-P9-UK | Filter FLT-P9-HEPA; its guide exists as a PDF |
| C70 | C70 Coffee Machine | HL-C70-UK | Error codes F2, F5, F8; descaling tablets DSC-1; no vinegar |
The rules Hollin's documents state
These are the facts the assistant must get right. They come from the current documents in the shared data.
| Rule | Value | Document |
|---|---|---|
| Warranty | 2 years on every appliance; 5 years on X200 and X300 Pro motors | pol-warranty-2026 |
| Not covered | Blade wear, cracked jars, limescale damage, accidental damage, use abroad with a converter | pol-warranty-2026, faq-abroad |
| Old warranty (archived, wrong today) | 1 year; motors 2 years; cracked jars covered | pol-warranty-2024 |
| Repair turnaround | Within 10 working days; 2 extra working days for repairs booked 1 to 16 October 2026 (3 after the 8 October update) | pol-repairs, ntc-repair-centre-move |
| Inspection fee | £25 for out-of-warranty repairs, deducted if you go ahead | pol-repairs |
| Spare parts delivery | £3.95, free over £30, 3 to 5 working days | pol-spare-parts |
| X200 jar recall | Batch B-2291: stop using, free replacement | ntc-x200-recall |
| Partner labour rates | Brook Street £42 an hour; Calder Fix £39 an hour; each sees only its own | ptn-brook-rates, ptn-calder-rates |
Dates
| Date | What happens (all simulated) |
|---|---|
| 10 February 2024 | Hollin publishes its 2024 warranty terms |
| 1 March 2026 | The 2026 warranty terms take effect. The 2024 page is archived but stays on the help centre |
| August 2026 | Kwame builds the prototype |
| 5 September 2026 | Ellie Shaw's X200 stops with E7 (case 30412) |
| 12 September 2026 | An agent quotes the archived motor cover (case 30501) |
| 14 September 2026 | X200 jar recall notice |
| 20 September 2026 | A forum post carrying an injected instruction appears |
| 21 September to 2 October 2026 | Six agents try the prototype; its query log is kept |
| 28 September 2026 | Repair centre move notice |
| 29 and 30 September 2026 | Calder Fix and Hollin exchange the pump claim email; Ellie's X200 shows E7 again and goes to Brook Street (case 30540) |
| Friday 2 October 2026 | The corpus snapshot in the shared data is exported (SNAPSHOT) |
| Monday 5 October 2026 | Grace hands you the project |
| Thursday 8 October 2026 | The update batch: one notice changed, bulletin SB-115 added, SB-114 withdrawn. At 10:07 Hannah withdraws SB-114 at the source (Module 5 Lesson 4) |
| Thursday 8 October 2026 (replay) | In a simulated replay, a re-index ships with the permission filter off (Module 12 Lesson 5) |
| Friday 9 October 2026 | The course date (COURSE_DATE). Ellie writes in to ask whether her E7 fault is covered; her chat is the example every module replays through Hollin Answers in development. Every lesson treats this day as "today" |
| Friday 9 October 2026, 09:55 and 10:00 | Bulletin SB-112 is sent to Dan; at 10:00 Hannah restricts SB-112 to staff while she revises it (Module 12 Lesson 4 and lab) |
| Monday 12 October 2026 | A load test replays "where is my repair" messages through a cache keyed on the message text (Module 7 incident file) |
| Wednesday 14 October 2026 | Ellie's repair (case 30540) is due back: 10 working days from 30 September (Module 10 Lesson 3) |
| Monday 2 November 2026 (planned) | Pilot for Hollin's support agents, if the Module 11 gate passes |
| Monday 30 November 2026 (planned) | Partners and the public help centre, if the capstone gauntlet passes |
Before 30 November no customer talks to Hollin Answers directly: customers' chats, Ellie's included, reach it only as replays through the service in development or staging.
What you build, module by module
| Module | The piece you add to Hollin Answers |
|---|---|
| 1. RAG Foundations | The first grounded answer with a citation, and a written decision on which source needs retrieval and which needs a database lookup |
| 2. Document Ingestion | Loaders for HTML, PDF, CSV, email and Markdown; normalisation; duplicate detection; a typed document record with access rights and provenance |
| 3. Chunking | A chunker that keeps headings and tables together, stable chunk IDs, and header paths injected into chunks |
| 4. Embeddings | A cached embedder with a model version on every vector, and the first retrieval scores on the labelled questions |
| 5. Vector Stores | A retriever interface with Chroma and FAISS behind it, permission filters applied inside the search, and the 8 October update applied as upserts and deletes |
| 6. Search | BM25, hybrid search (a weighted sum tuned on held-out questions, with reciprocal rank fusion as the fallback when nothing is labelled), thresholds, and current-over-archived ranking |
| 7. Query Understanding | Query rewriting (Ellie's "is it covered?"), routing between documents and the case database, and a cache of prepared questions |
| 8. Reranking | A cross-encoder reranker, duplicate collapse and a context assembler that fits the budget |
| 9. Generation | The answer contract: grounded answers, validated citations, an honest "I don't know", and the injected forum post contained |
| 10. Patterns | Corrective RAG and an agentic loop in LangGraph, text-to-SQL over the case table, compared against the plain pipeline |
| 11. Evaluation | The evaluation harness, the labelled questions extended, and a regression gate |
| 12. Production | Caching, freshness, permissions at every step, tracing and fallbacks |
| 13. Advanced | Mining the query log, detecting questions the corpus cannot answer, an audit of the corpus |
| 14. Capstone | The production service: FastAPI, PostgreSQL with pgvector, multi-tenant, streamed answers with citations, the gauntlet of probes in CI |
One repository: which module adds which file
Everything you keep goes in one repository, hollin-answers, with a single Python package called hollin/. Lessons run in Colab, so their code sits in cells; the module labs write the files. In Colab the repository lives in your Google Drive, which the Module 1 lab mounts, so it survives from one session to the next; each later lab writes an earlier module's file only when it is missing, as an exact copy of that module's file. Labs that must run in a temporary folder for testing create hollin/ there with the same layout and say so.
From Module 5 on, every index is built from the shared data (hollin/data.py) with chunks_of, which splits each document at blank lines, so each lab and the capstone run on their own. Modules 2 and 3 show how the production feed would be built (out/records.jsonl, out/chunks.jsonl); the Module 5 lab's Going further step indexes out/chunks.jsonl instead.
The table lists every file a lab writes, in the module that first writes it, and what the lab produces. All data in these files is simulated.
| Module | Adds to the hollin-answers repository |
|---|---|
| 1 | The package itself: pyproject.toml (every library the finished package imports, at course-guide/02's versions), .gitignore, hollin/llm.py (the setup cell, llm created once, and the ask() guard), hollin/data.py (the shared data from course-guide/06), hollin/naive.py (the first grounded answer), tests/conftest.py, tests/test_naive.py |
| 2 | hollin/ingest/: __init__.py (ingest), __main__.py (python -m hollin.ingest), loaders.py (one loader per format), normalise.py, dedupe.py, records.py (the typed DocumentRecord); tests/test_ingest.py. Outputs raw/, out/records.jsonl and out/rejections.jsonl |
| 3 | hollin/chunk/: __init__.py (chunk_record, chunk_all), records.py (the Chunk record), split.py (splitting, header paths, metadata, stable ids), __main__.py (python -m hollin.chunk); evals/chunk_recall.py; tests/test_chunk.py. Output out/chunks.jsonl |
| 4 | hollin/embeddings.py (the cached, versioned embedder), evals/retrieval.py (recall and MRR on the labelled questions), evals/embedding_decision.py and its result evals/results/embedding-decision.json, tests/test_embeddings.py. Builds index/embeddings.sqlite and index/qdrant (both ignored by Git) |
| 5 | hollin/stores/: __init__.py, base.py (the retriever interface), chroma_store.py, faiss_store.py, filters.py (permissions), sync.py (upserts, deletes and the gate); tests/test_stores.py. The blue/green pointer is index/active.json |
| 6 | hollin/search/: __init__.py (open_search), bm25.py, hybrid.py (fusion), signals.py (thresholds, recency, authority); evals/search_decision.py and evals/results/search-decision.json; tests/test_search.py |
| 7 | hollin/query/: rewrite.py, router.py (routing and the transformation cache, which stores prepared questions, not answers); evals/query_decision.py and evals/results/query-decision.json; tests/test_query.py |
| 8 | hollin/rerank.py (the reranker; current_first puts current documents ahead of archived ones before reranking), hollin/context.py; tests/test_context.py |
| 9 | hollin/answer.py (the answer contract and citation check), hollin/grounding.py (rule checks and the local NLI check), hollin/guard.py (untrusted content); tests/fakes.py, tests/test_answer.py |
| 10 | hollin/patterns/: crag.py, agent.py (LangGraph), sql.py, conversation.py (conversational RAG); tests/scripted.py, tests/test_patterns.py |
| 11 | evals/: questions.py (52 cases), harness.py, judge.py, gate.py, sets/full-v1.jsonl, sets/nightly-v1.jsonl, results/baseline-pr.json, results/judge-calibration.json; tests/test_evals.py. Lesson 5's workflow becomes Module 14's ci.yml |
| 12 | hollin/cache.py, hollin/trace.py, hollin/fallback.py, hollin/governance.py (access checks at every hop and the audit chain); tests/test_production.py. Writes logs/traces.jsonl (.gitignore gains logs/) |
| 13 | hollin/insights.py (log mining, unanswerable detection, corpus audit); tests/test_insights.py; training/triplets.jsonl (committed) and evals/results/audit-2026-10-09.json |
| 14 | hollin/api/ (the FastAPI service: __init__.py, app.py, pipeline.py, which runs Module 7's free query fixes first, __main__.py), hollin/stores/pgvector_store.py, tests/gauntlet/ (__init__.py, conftest.py, test_access.py, test_attacks.py, test_freshness.py, test_failures.py), .github/workflows/ci.yml, Dockerfile, requirements.txt (the service-only pins), docker-compose.yml, evals/results/service/baseline-pr.json, and a README whose last section is the write-up |
The finished product
- Ingestion from five formats into typed records that carry access rights and provenance
- Structure-aware chunks with stable IDs
- Local embeddings (
BAAI/bge-small-en-v1.5) with a model version on every vector - Hybrid search (BM25 and dense, fused), a cross-encoder reranker, thresholds that can return nothing
- Permission filters applied inside retrieval, per user and per partner
- Grounded answers with verified citations and an honest "I don't know"
- Retrieved text treated as untrusted data
- Routing to the case database for live facts
- An evaluation harness and a regression gate in CI
- Caching, freshness handling, tracing and fallbacks
- A FastAPI service on PostgreSQL with pgvector, run with Docker Compose
What was kept from the earlier version of this course
The earlier course was set at ShopSphere, a made-up online shop. For this version:
- Kept: the planted-trap corpus design (an archived policy that contradicts the current one, a ticket quoting the old policy, a near-duplicate, one error code with two meanings on two blenders, rare names and part numbers, a safety recall, no-answer questions), the X200 and X300 blenders and their error codes, the split between documents (retrieved) and orders or cases (looked up), plain-Python-first builds, and the capstone's multi-tenant service and probe gauntlet.
- Changed: the company is now a manufacturer's after-sales service, not a shop, so it does not overlap with BrightCart in the AI Engineering and Agentic AI courses. One currency (pounds) replaces a mix of dollars and rupees. The thin cast became a fixed cast. The second tenant (previously a store called BrightMart) became two repair partners. The corpus gained structured documents (a Markdown guide with tables, a PDF page, a CSV, an email thread, a web page with boilerplate), access rules, an injected instruction, a query log and an update batch, so every module works on the same data.