Part 8 · Module 1: RAG Foundations

Lesson 2: Nine stages, and where answers quietly go wrong

25 min read·13 Oct 2026

Module 1 · Lesson 2 · About 50 minutes · Needs: Lesson 1 (what retrieval adds to a model) and basic Python

The situation

On 12 September 2026 a Hollin agent told a customer the X300 Pro motor was covered for 2 years; the current terms say 5 (case 30501). When an agent asked Kwame's prototype the same thing, the answer got a thumbs down.

Grace asks: "Which part is broken?" A wrong answer is a symptom that can start in any of nine stages. You build all nine, trace one question through them, q02, "how many years is the X300 Pro motor covered for", and add checks that say where an answer went wrong. The Hollin data is simulated.

A real case, first

In May 2024 a Stanford team, Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher Manning and Daniel Ho, tested three commercial legal research tools built on retrieval ("Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools", arXiv, 2024), whose vendors claimed retrieval avoids made-up answers. They ran 202 preregistered queries and checked every answer by hand.

In the Stanford study (Magesh and others, 2024) all three legal tools built on retrieval still hallucinated on 17% to 33% of queries (green: accurate, grey: incomplete, red: hallucinated).

The tools "each hallucinate between 17% and 33% of the time". Lexis+ AI was accurate on 65% of queries, Westlaw AI-Assisted Research on 42%.

A misgrounded answer carries a citation, but the source is misread or does not apply, so it looks checked when it is not (red: failure, grey: incomplete).

"A response is considered hallucinated if it is either incorrect or misgrounded." Misgrounded means cited, but the source is misread or does not apply; the authors call this "potentially more dangerous than fabricating a case outright". They traced failures to poor retrieval as well as to reasoning. The products may have changed since 2024.

What you will be able to do

Build the nine stages and trace a question through them.
Add a check per stage for its silent failure.
Time and price each stage.
Sort a bad answer by stage and by where it was lost.
Explain why ten test questions prove little.

1. The nine pipeline stages (Essential)

To fix a wrong answer you must know which step to open. That needs steps that are separate, named and recorded.

Four indexing stages run when documents change; five query stages run per question, and only generate calls the model (blue: your code, amber: the model).

A retrieval system is two pipelines. The indexing pipeline runs when documents change: parse, chunk, embed, index. The query pipeline runs for every question: retrieve, rerank, assemble, generate, cite. Each is a pipeline stage: one job, one output you can inspect. Modules 2 to 9 improve them; here each is as simple as possible. Only stage 8 calls the model. First, the standard setup.

python

%pip install -q -U langchain langgraph langchain-groq langchain-google-genai langchain-ollama

python

import os
from langchain.chat_models import init_chat_model

PROVIDER = "groq"   # "groq" or "gemini" in Colab; "ollama" on your own computer
MODELS = {"groq": "groq:openai/gpt-oss-20b", "gemini": "google_genai:gemini-3.5-flash-lite", "ollama": "ollama:llama3.2:3b"}
KEY = {"groq": "GROQ_API_KEY", "gemini": "GOOGLE_API_KEY", "ollama": None}[PROVIDER]
if KEY and KEY not in os.environ:
    from google.colab import userdata   # keys live in Colab's Secrets panel
    os.environ[KEY] = userdata.get(KEY)
llm = init_chat_model(MODELS[PROVIDER], temperature=0, max_retries=6)

Code walkthrough: the setup cell from course-guide/02. Change PROVIDER to switch; max_retries=6 retries on "too many requests".

python

def ask(model, messages):
    """Returns (reply, None), or (None, reason) after two refusals."""
    for attempt in range(2):                     # one retry: the next reply is usually fine
        try:
            return model.invoke(messages), None
        except Exception as err:
            last = err
    return None, f"provider_error: {type(last).__name__}"

Code walkthrough: the course's guard. After two rejected replies it returns None and a provider_error reason for the next stage to handle.

The data: five document blocks from course-guide/06.

python

%pip install -q sentence-transformers==6.1.0

python

from datetime import date

COURSE_DATE = date(2026, 10, 9)   # "today" in every lesson; never use date.today()
SNAPSHOT = date(2026, 10, 2)      # the day the corpus below was exported

SOURCE = {"policy": "help-centre", "faq": "help-centre", "notice": "help-centre",
          "manual": "manuals", "bulletin": "service-portal", "ticket": "ticket-system",
          "partner": "partner-portal", "internal": "finance-share", "forum": "community-forum"}
AUTHORITY = {"policy": 1.0, "notice": 1.0, "manual": 0.9, "bulletin": 0.9, "partner": 0.8,
             "internal": 0.8, "faq": 0.7, "ticket": 0.3, "forum": 0.1}

def doc(id, type, title, updated, text, acl=("public",), products=(), status="current", authority=None):
    """One document record. acl lists the groups allowed to see it."""
    return {"id": id, "type": type, "title": title, "updated": updated, "text": text,
            "acl": list(acl), "products": list(products), "status": status,
            "source": SOURCE[type], "authority": AUTHORITY[type] if authority is None else authority}

DOCS = []

Code walkthrough: doc(...) builds one record; this lesson uses status and authority.

python

DOCS += [
    doc("pol-warranty-2026", "policy", "Warranty terms (2026)", "2026-03-01",
        "Every Hollin appliance has a 2-year warranty from the date of purchase. Blender motors on the X200 "
        "and X300 Pro are covered for 5 years. The warranty covers manufacturing faults. It does not cover "
        "blade wear, cracked jars, limescale damage or accidental damage. You need proof of purchase to claim."),
    doc("pol-warranty-2024", "policy", "Warranty terms (2024, archived)", "2024-02-10",
        "Every Hollin appliance has a 1-year warranty from the date of purchase. Blender motors are covered "
        "for 2 years. The warranty covers manufacturing faults and cracked jars. You need proof of purchase "
        "to claim.", status="archived"),
    doc("pol-repairs", "policy", "Repairs and turnaround", "2026-05-12",
        "We aim to repair and return an appliance within 10 working days of receiving it. Collection by "
        "courier is free for appliances under warranty. Out-of-warranty repairs carry a £25 inspection fee, "
        "which is deducted from the repair price if you go ahead."),
    doc("pol-spare-parts", "policy", "Ordering spare parts", "2026-04-20",
        "Spare parts are sold at parts.hollin.example. Standard delivery takes 3 to 5 working days and costs "
        "£3.95. Delivery is free on parts orders over £30."),
    doc("pol-privacy", "policy", "Your account data", "2026-02-01",
        "You can download or delete your account data from Settings, then Privacy. Deletion requests are "
        "completed within 30 days."),
    doc("pol-registration", "policy", "Registering your appliance", "2025-11-02",
        "Register your appliance at hollin.example/register so we can contact you about safety notices. "
        "You do not need to register to claim under the warranty."),
]

Code walkthrough: six policies; the archived pol-warranty-2024 says motors get 2 years.

python

DOCS += [
    doc("man-x200-specs", "manual", "X200 Blender specifications", "2025-06-01",
        "The X200 blender has a 1,200 W motor, a 2-litre Tritan jar and three speeds. High speed reaches "
        "22,000 RPM. Model number HL-X200-UK.", products=["X200"]),
    doc("man-x200-errors", "manual", "X200 error codes", "2025-06-01",
        "E4 means the jar lid is not locked: twist the lid until it clicks. E7 means the motor has overheated: "
        "unplug the blender for 20 minutes. E9 means a sensor fault: contact Hollin support.", products=["X200"]),
    doc("man-x200-cleaning", "manual", "X200 cleaning and care", "2025-06-01",
        "Wash the jar and blades in warm soapy water or in the dishwasher. Never put the motor base in water; "
        "wipe it with a damp cloth.", products=["X200"]),
    doc("man-x200-blades", "manual", "X200 blade assembly", "2025-06-01",
        "A grinding noise, or food left in chunks, usually means the blades are worn. Replace the blade "
        "assembly, part BLD-X200, about once a year with daily use.", products=["X200"]),
    doc("man-x300-specs", "manual", "X300 Pro Blender specifications", "2026-04-10",
        "The X300 Pro blender has a 1,500 W motor, a vacuum blending mode and ten speeds. High speed reaches "
        "30,000 RPM. Model number HL-X300-UK.", products=["X300"]),
    doc("man-x300-errors", "manual", "X300 Pro error codes", "2026-04-10",
        "E4 means the vacuum lid seal is loose: take the seal out and press it back into its groove. E12 means "
        "the vacuum pump has failed: contact Hollin support.", products=["X300"]),
    doc("man-k50-kettle", "manual", "K50 Smart Kettle", "2025-09-09",
        "The K50 smart kettle boils 1.7 litres in about four minutes and can keep water warm for up to two "
        "hours. Descale it once a month with citric acid.", products=["K50"]),
    doc("man-k50-app", "manual", "Pairing the K50 with the Hollin Home app", "2025-09-09",
        "To pair the K50 with the Hollin Home app, hold the temperature dial down for 5 seconds until the ring "
        "light flashes white, then choose Add appliance in the app.", products=["K50"]),
]

Code walkthrough: eight manual pages.

python

DOCS += [
    doc("faq-warranty-claim", "faq", "How do I make a warranty claim?", "2026-03-01",
        "Book a repair at hollin.example/repair with your model number and proof of purchase. A courier "
        "collects the appliance from your door. Most repairs are back with you within 10 working days."),
    doc("faq-warranty-claim-copy", "faq", "Making a warranty claim", "2026-03-01",
        "Book a repair at hollin.example/repair with your model number and proof of purchase. A courier "
        "collects the appliance from your door. Most repairs are back with you within 10 working days!",
        authority=0.6),
    doc("faq-proof-of-purchase", "faq", "What counts as proof of purchase?", "2026-03-01",
        "A receipt, an order confirmation email or a bank statement that shows the retailer, the product "
        "and the date. A photo of the receipt is fine."),
    doc("faq-abroad", "faq", "Using Hollin appliances abroad", "2025-11-02",
        "Hollin appliances are made for UK 230 V sockets. Using one with a voltage converter abroad is not "
        "covered by the warranty."),
    doc("faq-vat-invoice", "faq", "VAT invoices for spare parts", "2026-04-20",
        "Business customers can ask for a VAT invoice for spare parts. Add your VAT number at checkout, or "
        "email parts@hollin.example with your order number."),
    doc("faq-hard-water", "faq", "Hard water and limescale", "2025-09-09",
        "In hard-water areas, descale kettles and coffee machines more often. Limescale damage is not "
        "covered by the warranty."),
    doc("faq-repair-tracking", "faq", "Tracking a repair", "2026-05-12",
        "We email a tracking link when the courier collects your appliance and again when it is on its way "
        "back. A repair shows as Booked, In workshop, Repaired, then Dispatched."),
]

Code walkthrough: seven help-centre answers.

python

PARTNERS_AND_STAFF = ("partners", "staff")
DOCS += [
    doc("sb-109", "bulletin", "SB-109: X200 motor coupling", "2026-01-15",
        "Some X200 blenders made before March 2025 have a soft motor coupling that wears early. Fit coupling "
        "kit CPL-X200 at the first repair visit, free of charge.", acl=PARTNERS_AND_STAFF, products=["X200"]),
    doc("sb-112", "bulletin", "SB-112: X300 Pro pump replacement", "2026-06-30",
        "If E12 returns after the seal has been reseated, replace the vacuum pump, part PMP-X300. Allow 35 "
        "minutes of labour.", acl=PARTNERS_AND_STAFF, products=["X300"]),
    doc("sb-114", "bulletin", "SB-114: K50 firmware 2.3", "2026-09-22",
        "K50 kettles on firmware 2.1 may switch off before boiling. Update them to firmware 2.3 through the "
        "Hollin Home app.", acl=PARTNERS_AND_STAFF, products=["K50"]),
    doc("ntc-x200-recall", "notice", "Safety notice: X200 jar recall", "2026-09-14",
        "X200 jars from batch B-2291 may crack when used with hot liquids. Stop using the jar and order a "
        "free replacement at hollin.example/recall. The batch number is printed on the base of the jar.",
        products=["X200"]),
    doc("ntc-repair-centre-move", "notice", "Repair centre move", "2026-09-28",
        "Our repair centre is moving from Cleckheaton to Wakefield. Repairs booked between 1 and 16 October "
        "2026 may take 2 extra working days."),
]

Code walkthrough: three bulletins, two notices.

python

DOCS += [
    doc("tkt-30412", "ticket", "Case 30412: X200 stopped mid-blend", "2026-09-05",
        "Case 30412. Customer Ellie Shaw reports her X200 stopped mid-blend showing E7. Agent advised unplugging for 20 "
        "minutes; the blender worked afterwards.", acl=["staff"], products=["X200"]),
    doc("tkt-30455", "ticket", "Case 30455: K50 will not pair", "2026-09-10",
        "Case 30455. The K50 would not pair with an older Android phone. Holding the dial and the power button together "
        "for 15 seconds reset it, and pairing then worked.", acl=["staff"], products=["K50"]),
    doc("tkt-30501", "ticket", "Case 30501: motor cover question", "2026-09-12",
        "Case 30501. Agent told the customer the X300 Pro motor is covered for 2 years. Note from team lead: this followed "
        "the archived 2024 warranty terms; the current cover is 5 years.", acl=["staff"], products=["X300"]),
    doc("ptn-handbook", "partner", "Service partner handbook", "2026-07-01",
        "Submit warranty labour claims within 14 days of the repair, quoting the Hollin case number. Use "
        "genuine Hollin parts only. Hollin pays labour at the hourly rate in your partner agreement.",
        acl=PARTNERS_AND_STAFF),
    doc("ptn-brook-rates", "partner", "Brook Street Repairs: partner agreement", "2026-04-01",
        "Hollin pays Brook Street Repairs £42 per hour of warranty labour and £18 per home visit. The "
        "agreement runs to 31 March 2027.", acl=["partner:brook-street", "staff"]),
    doc("ptn-calder-rates", "partner", "Calder Fix: partner agreement", "2026-07-01",
        "Hollin pays Calder Fix £39 per hour of warranty labour and £20 per home visit. The agreement runs "
        "to 30 June 2027.", acl=["partner:calder-fix", "staff"]),
    doc("int-parts-cost", "internal", "Parts cost prices (internal)", "2026-08-01",
        "Internal cost prices: X200 jar £6.10, BLD-X200 blade assembly £3.40, PMP-X300 pump £11.80, "
        "FLT-P9-HEPA filter £7.25. Do not share outside Finance.", acl=["finance"]),
]
print(len(DOCS), "documents")

Code walkthrough: tickets and partner files; tkt-30501 holds both "2 years" and "5 years". Prints 33 documents. Access rights wait for Module 5.

python

QUESTIONS = [
    ("q01", "can I put the blender base in the sink", {"man-x200-cleaning"}, "No. Never put the motor base in water; wipe it with a damp cloth.", "paraphrase"),
    ("q02", "how many years is the X300 Pro motor covered for", {"pol-warranty-2026"}, "5 years.", "stale"),
    ("q03", "the smoothie maker stopped and felt hot", {"man-x200-errors", "tkt-30412"}, "E7: the motor overheated. Unplug it for 20 minutes.", "paraphrase"),
    ("q04", "how do I get rid of limescale in the kettle", {"man-k50-kettle", "faq-hard-water"}, "Descale it once a month with citric acid.", "paraphrase"),
    ("q05", "how long until I get my appliance back from repair", {"pol-repairs", "faq-warranty-claim", "faq-warranty-claim-copy", "ntc-repair-centre-move"}, "Within 10 working days; repairs booked 1 to 16 October 2026 may take 2 extra working days.", "paraphrase"),
    ("q06", "what can I use to show when I bought it", {"faq-proof-of-purchase"}, "A receipt, order confirmation email or bank statement showing retailer, product and date.", "paraphrase"),
    ("q07", "can I use my kettle on holiday in America", {"faq-abroad"}, "They are made for UK 230 V sockets; use with a converter abroad is not covered.", "paraphrase"),
    ("q09", "E4", {"man-x200-errors", "man-x300-errors"}, "X200: the jar lid is not locked. X300 Pro: the vacuum lid seal is loose.", "code"),
    ("q10", "E12", {"man-x300-errors", "sb-112"}, "The X300 Pro vacuum pump has failed: contact Hollin support.", "code"),
    ("q12", "HL-X300-UK", {"man-x300-specs"}, "The model number of the X300 Pro blender.", "code"),
    ("q13", "batch B-2291", {"ntc-x200-recall"}, "Recalled X200 jars: stop using them and order a free replacement.", "code"),
    ("q15", "case 30455", {"tkt-30455"}, "A K50 that would not pair; holding the dial and power button for 15 seconds reset it.", "restricted"),
    ("q16", "Tritan", {"man-x200-specs"}, "The X200's 2-litre jar is made of Tritan.", "name"),
    ("q17", "Cleckheaton", {"ntc-repair-centre-move"}, "The repair centre is moving from Cleckheaton to Wakefield.", "name"),
    ("q19", "what does E7 mean on my X200", {"man-x200-errors", "tkt-30412"}, "The motor has overheated: unplug it for 20 minutes.", "mixed"),
    ("q20", "X300 top speed", {"man-x300-specs"}, "30,000 RPM.", "mixed"),
    ("q21", "how do I connect the K50 to my phone", {"man-k50-app", "tkt-30455"}, "Hold the dial for 5 seconds until the ring flashes white, then Add appliance in the Hollin Home app.", "mixed"),
    ("q23", "is blade wear covered for the X200", {"pol-warranty-2026"}, "No. Blade wear is not covered.", "mixed"),
    ("q27", "how much does delivery of spare parts cost", {"pol-spare-parts"}, "£3.95, free on parts orders over £30.", "natural"),
    ("q28", "how do I delete my account data", {"pol-privacy"}, "Settings, then Privacy; deletion is completed within 30 days.", "natural"),
    ("q29", "can I get a VAT invoice for spare parts", {"faq-vat-invoice"}, "Yes: add your VAT number at checkout or email parts@hollin.example.", "natural"),
    ("q30", "how much is the inspection fee for an out-of-warranty repair", {"pol-repairs"}, "£25, deducted from the repair price if you go ahead.", "natural"),
    ("q31", "is a cracked X200 jar covered, and what if it is from batch B-2291", {"pol-warranty-2026", "ntc-x200-recall"}, "Not under the warranty, but jars from batch B-2291 are replaced free under the recall.", "multi-hop"),
    ("q33", "do you sell electric scooters", set(), "", "no-answer"),
    ("q34", "what is the capital of Peru", set(), "", "no-answer"),
    ("q35", "does the X900 drone have GPS", set(), "", "no-answer"),
    ("q36", "how do I descale the X300 Pro", set(), "", "no-answer"),
]
print(len(QUESTIONS), "labelled questions")

Code walkthrough: the 27 labelled questions whose documents are in this slice: ID, question, right documents, answer, type.

Hollin's 33 documents become 33 chunks and 33 vectors of 384 numbers in about a tenth of a second on a laptop CPU.

Stages 1 to 4 build an index, a searchable table of vectors. Documents are 18 to 50 words, so each is one chunk, keeping status and authority for later checks.

python

import re, time, numpy as np
from sentence_transformers import SentenceTransformer, CrossEncoder

def parse(d):                          # 1 parse: raw document to clean text
    return re.sub(r"\s+", " ", d["text"]).strip()

def chunk(d, text):                    # 2 chunk: short documents stay whole
    return [{"id": d["id"], "text": text, "status": d["status"], "authority": d["authority"]}]

embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
CHUNKS = [c for d in DOCS for c in chunk(d, parse(d))]
start = time.perf_counter()
VECTORS = embedder.encode([c["text"] for c in CHUNKS], normalize_embeddings=True)   # 3 embed
INDEX = {"vectors": VECTORS, "chunks": CHUNKS}                                      # 4 index
BY_ID = {c["id"]: c for c in CHUNKS}
print(len(DOCS), "documents ->", len(CHUNKS), "chunks ->", VECTORS.shape,
      f"in {time.perf_counter() - start:.2f} s")

Code walkthrough

parse tidies spaces; chunk keeps each document whole.
SentenceTransformer(...) downloads a 130 MB embedding model; encode gives 384 unit-length numbers per chunk, so a dot product is cosine similarity.
INDEX pairs vectors with chunks; BY_ID looks up a chunk. Prints (33, 384) and the time.

For q02 both retrieval and the reranker put ticket tkt-30501 (red, a weak source) first and the current terms (green) second.

Stage 5, retrieve, keeps the ten chunks closest to the question. Stage 6, rerank, reads question and chunk together with a slower, sharper cross-encoder and keeps three. Both put ticket tkt-30501 first and the current terms second.

python

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

def retrieve(question, k=10):          # 5 retrieve: the k closest chunks
    scores = INDEX["vectors"] @ embedder.encode(question, normalize_embeddings=True)
    return [(INDEX["chunks"][i]["id"], round(float(scores[i]), 3)) for i in np.argsort(-scores)[:k]]

def rerank(question, hits, n=3):       # 6 rerank: read question and chunk together, keep n
    scores = reranker.predict([(question, BY_ID[i]["text"]) for i, _ in hits])
    pairs = [(i, round(float(s), 2)) for (i, _), s in zip(hits, scores)]
    return sorted(pairs, key=lambda p: -p[1])[:n]

Q02 = QUESTIONS[1][1]
print(retrieve(Q02)[:5])
print(rerank(Q02, retrieve(Q02), n=5))

Code walkthrough

CrossEncoder(...) downloads a 90 MB reranker; its scores are unbounded, above 0 roughly means relevant.
retrieve scores every chunk with one matrix product; rerank scores pairs and keeps n.
Prints: tkt-30501 (0.939, 10.61), then pol-warranty-2026 (0.794, 9.24).

The assembled prompt is an instruction, three labelled chunks using 106 of the 120-word budget, and the question.

Stage 7, assemble, builds the prompt: instruction, labelled chunks, question, within a budget (106 of 120 words here). Order matters: models do best with information "at the beginning or end of the input context" (Liu and others, Lost in the Middle, 2023).

python

def assemble(question, ranked, budget=120):   # 7 assemble: labelled blocks within a word budget
    blocks, used, words = [], [], 0
    for i, _ in ranked:
        n_words = len(BY_ID[i]["text"].split())
        if words + n_words > budget:
            break                              # this chunk and everything after it is dropped
        blocks.append(f"[{i}] {BY_ID[i]['text']}")
        used.append(i)
        words += n_words
    prompt = ("Answer from the sources only. Cite each claim with its source ID in square brackets, "
              "like [pol-repairs]. If the sources do not answer, say you don't know.\n\n"
              + "\n\n".join(blocks) + f"\n\nQuestion: {question}")
    return prompt, used

prompt, used = assemble(Q02, rerank(Q02, retrieve(Q02)))
print(used, len(prompt.split()), "words in the prompt")
print(prompt[:200])

Code walkthrough

The loop adds chunks in rank order until one does not fit; used records what went in.
Prints the three IDs and 147 words in the prompt.

run() keeps a trace of what each stage produced for q02, so a bad answer can be followed back to the stage that caused it.

Generate sends the prompt through ask. Cite checks every [id] names a chunk the model was given. run chains stages 5 to 9 into a trace: each stage's output and time.

python

def generate(prompt):                  # 8 generate: the one stage that calls the model
    reply, reason = ask(llm, prompt)
    if reply is None:
        return "", reason, {}
    return reply.text or "", None if reply.text else "empty_reply", reply.usage_metadata or {}

def cite(answer, used):                # 9 cite: every [id] must be a chunk the model was given
    cited = re.findall(r"\[([a-z0-9-]+)\]", answer)
    return {"cited": cited, "unknown": [c for c in cited if c not in used]}

def run(question, k=10, n=3, budget=120, answer=True):
    t, ms, clock = {"question": question}, {}, time.perf_counter()
    def lap(stage):
        nonlocal clock
        ms[stage] = round((time.perf_counter() - clock) * 1000, 1)
        clock = time.perf_counter()
    t["retrieved"] = retrieve(question, k); lap("retrieve")
    t["reranked"] = rerank(question, t["retrieved"], n); lap("rerank")
    t["prompt"], t["used"] = assemble(question, t["reranked"], budget); lap("assemble")
    t["answer"], t["reason"], t["usage"] = generate(t["prompt"]) if answer else ("", "not asked", {}); lap("generate")
    t["cite"] = cite(t["answer"], t["used"]); lap("cite")
    t["ms"] = ms
    return t

trace = run(Q02)
print(trace["answer"] or trace["reason"])
print(trace["cite"])

Code walkthrough

generate returns text, a reason if there is none (provider_error or empty_reply), and the provider's token counts.
cite lists cited IDs and any not in the prompt.
lap records milliseconds per stage; answer=False skips the model.
One model call. Look for 5 years citing [pol-warranty-2026]; after a provider failure you see the reason and later blocks still run.

2. Where each stage silently fails (Essential)

A crash tells you something broke. Most faults do not crash: each stage passes on something plausible.

Every stage has a silent failure, an output that is worse but raises no error, with a Hollin example of each (red: the failure).

That is a silent failure: worse output, no error. The prototype's log shows several: finance-only cost prices indexed by mistake, withdrawn bulletin sb-114 never removed, tkt-30501 above the current terms.

One cheap check per stage turns q02's hidden problem, a weak ticket first in the prompt, into a warning (green: passed, red: warning).

A cheap check per stage turns silent into visible. On q02 one fires: the ticket, authority 0.3, is first in the prompt.

python

def stage_checks(t):
    warn = []
    if len(INDEX["chunks"]) != len(DOCS):
        warn.append(f"index: {len(INDEX['chunks'])} chunks for {len(DOCS)} documents")
    if any(not BY_ID[i]["text"] for i, _ in t["retrieved"]):
        warn.append("parse: a retrieved chunk is empty")
    if t["reranked"][0][1] < 0:
        warn.append(f"rerank: best score {t['reranked'][0][1]} is below 0")
    for i in t["used"]:
        if BY_ID[i]["status"] != "current" or BY_ID[i]["authority"] < 0.5:
            warn.append(f"assemble: {i} is {BY_ID[i]['status']}, authority {BY_ID[i]['authority']}")
    if len(t["used"]) < len(t["reranked"]):
        warn.append(f"assemble: budget dropped {len(t['reranked']) - len(t['used'])} chunk(s)")
    if t["answer"] and (t["cite"]["unknown"] or not t["cite"]["cited"]):
        warn.append(f"cite: unknown {t['cite']['unknown']} or no citation")
    return warn or ["no warnings"]

print(stage_checks(trace))

Code walkthrough

Each if checks one stage: index count, empty text, weak best score, stale or weak chunk, budget drops, citations.
On q02 it prints assemble: tkt-30501 is current, authority 0.3.

Break it: lose three documents from the index

With 30 of 33 chunks indexed, retrieval still returns ten confident results; only the index count check notices.

Say the embedding job times out after 30 of 33 chunks. Nothing errors.

python

saved = dict(INDEX)
INDEX["chunks"], INDEX["vectors"] = CHUNKS[:30], VECTORS[:30]   # the last three never arrive
broken = run("what does Hollin pay for a PMP-X300 pump", answer=False)
print(broken["retrieved"][:2])
print(stage_checks(broken))
INDEX.update(saved)                                              # put the index back

Code walkthrough

CHUNKS[:30] loses two partner agreements and the cost prices.
Retrieval still returns ten results; only the index check notices: index: 30 chunks for 33 documents. The last line restores the index.

3. Latency and cost budget per stage (Recommended)

Owen asks what an answer costs; Grace asks how long agents wait. Both are answered per stage.

Local stages take milliseconds on a laptop CPU; the model call (amber, dashed: measure it on your run) is usually the largest share of the wait.

On the course laptop's CPU, retrieval took about 6 ms, reranking about 10 ms, assembly and citing under 1 ms; Colab differs. Check generate on your run: the model call usually outweighs every local stage, so each stage gets a latency budget.

python

for stage, ms in trace["ms"].items():
    print(f"{stage:9} {ms:8.1f} ms")

PRICE_IN = {"groq": 0.075, "gemini": 0.30}   # US$ per million input tokens, course-guide/02
n_in = len(embedder.tokenizer(trace["prompt"])["input_ids"])
print(n_in, "prompt tokens, counted with the embedding model's tokenizer")
for name, price in PRICE_IN.items():
    print(f"{name}: ${n_in * price / 1e6 * 1000:.4f} of input per 1,000 questions")
print("provider's own count:", trace["usage"])

Code walkthrough

The loop prints each stage's milliseconds.
PRICE_IN: input prices from course-guide/02 (checked 9 October 2026).
embedder.tokenizer gives an estimate; each model tokenises differently. The last line prints the provider's own count, the one to bill by.

The q02 prompt is 234 tokens: about 1.8 US cents of input per 1,000 questions on Groq and 7 cents on Gemini, at course-guide/02 prices.

234 tokens is about 1.8 US cents of input per 1,000 questions on Groq, 7 on Gemini. Output tokens cost more and reasoning models add hidden ones, so read usage. The levers: fewer, better chunks; small models where they suffice.

4. Retrieval, assembly and generation failures (Essential)

"The bot was wrong" cannot be fixed. "Retrieved but cut by the budget" can, in one line.

Three questions, asked in order, sort a wrong answer into a retrieval, assembly or generation failure (red: where it failed).

Ask in order. Not retrieved: a retrieval failure. Retrieved but not in the prompt: an assembly failure. In the prompt but the answer is wrong: a generation failure.

python

def diagnose(t, relevant, expected):
    if not relevant & {i for i, _ in t["retrieved"]}:
        return "retrieval failure"
    if not relevant & set(t["used"]):
        return "assembly failure"
    if not t["answer"]:
        return f"no answer ({t.get('reason')})"
    if expected.lower() not in t["answer"].lower():
        return "generation failure"
    return "ok"

RELEVANT, EXPECTED = QUESTIONS[1][2], "5 years"
print("this run:", diagnose(trace, RELEVANT, EXPECTED))

Code walkthrough

& is set intersection: is a right document retrieved, then in used?
An empty answer reports its reason, not a model fault. The last test is crude; Module 11 builds better ones.

The same wrong answer to q02 has three different causes, and three different fixes, depending on k and the budget.

Break q02 three ways. k=1 keeps only the ticket. A 40-word budget fits the ticket, not the terms. With everything right, a "2 years" answer is the model's fault.

python

cases = {
    "k=1": (run(Q02, k=1, answer=False), "The motor is covered for 2 years. [tkt-30501]"),
    "budget=40": (run(Q02, budget=40, answer=False), "The motor is covered for 2 years. [tkt-30501]"),
    "full pipeline": (run(Q02, answer=False), "The motor is covered for 2 years. [tkt-30501]"),
}
for name, (t, scripted) in cases.items():
    t["answer"] = scripted                     # a scripted wrong answer: no model call
    print(f"{name:14} used={t['used']} -> {diagnose(t, RELEVANT, EXPECTED)}")

Code walkthrough

Each case changes one setting and takes the same scripted answer, with no model call. Output: retrieval failure, assembly failure, generation failure.

5. Four places the answer disappears (Essential)

Before blaming the model, check the answer ever reached it. It can vanish in four places.

Before the model sees anything, the answer can vanish in four places, checked in this order (grey: never existed, red: lost on the way).

Missing content: no document has it. Extraction failure: parsing lost it. Missed top-k: ranked below the cut-off. Lost in reranking: retrieved, then pushed out.

python

def where_did_it_go(question, relevant, k=10, n=3, use_rerank=True):
    if not relevant:
        return "missing content"
    if not any(BY_ID[i]["text"] for i in relevant):
        return "extraction failure"
    ids = [i for i, _ in retrieve(question, len(INDEX["chunks"]))]
    best = min(ids.index(i) + 1 for i in relevant)
    if best > k:
        return f"missed top-k (rank {best}, k={k})"
    kept = [i for i, _ in rerank(question, retrieve(question, k), n)] if use_rerank else ids[:k]
    if not relevant & set(kept):
        return f"lost in reranking (retrieved at rank {best})"
    return f"reached the prompt (rank {best})"

Code walkthrough

Checks run in funnel order and stop at the first failure.
Ranking every chunk gives best, the true rank, even below k.
use_rerank=False copies Kwame's prototype: top 3, no reranker.

Hollin has a live example of each place, from the prototype's log and the labelled questions; extraction is made in the break-it.

"How to descale X300" has no answer. "Cracked jar covered?", from the prototype's log, ranks the current terms 8th, below its cut of 3. q07's page is 3rd, then reranked out.

python

probes = [("how to descale X300", set(), {}),
          ("cracked jar covered?", {"pol-warranty-2026"}, {"k": 3, "use_rerank": False}),
          (QUESTIONS[6][1], QUESTIONS[6][2], {})]
for question, relevant, settings in probes:
    print(f"{question[:30]:30} -> {where_did_it_go(question, relevant, **settings)}")

from collections import Counter
print(Counter(where_did_it_go(q, rel).split(" (")[0] for _, q, rel, _, _ in QUESTIONS))

Code walkthrough

The cracked-jar probe uses the prototype's settings; its label is added in this lesson.
All 27 questions: reached the prompt: 22, missing content: 4, lost in reranking: 1.

Break it: a page with no text layer

An empty scanned page makes q20 return a ticket instead of the X300 specification, and nothing raises an error.

Say the X300 specification arrives as a scan with no text. It is embedded anyway.

python

spec = BY_ID["man-x300-specs"]
pos = CHUNKS.index(spec)
original = spec["text"]
spec["text"] = ""                                           # the scan gave no text
VECTORS[pos] = embedder.encode("", normalize_embeddings=True)
print(retrieve(QUESTIONS[15][1])[:2], "->", where_did_it_go(QUESTIONS[15][1], QUESTIONS[15][2]))
spec["text"] = original                                     # restore the page
VECTORS[pos] = embedder.encode(original, normalize_embeddings=True)

Code walkthrough

The chunk is emptied and re-embedded. q20's top result becomes tkt-30501 (0.666): extraction failure. The last lines restore it.

6. Wrong but fluent answers (Essential)

A crash costs a retry; "I don't know" a hand-over. Wrong and confident costs whatever the customer does next.

A crash and an "I don't know" are noticed at once; a wrong but fluent answer often is not, so it costs the most.

A wrong but fluent answer reads well and goes unnoticed. Case 30501 was one. Misgrounded answers add a citation that makes them look checked.

A stale answer passes the citation check (green) and fails the currency check (red), so both are needed.

Stage 9 only asks whether the cited ID was given, so a misgrounded answer passes. Also ask: is it current and trusted?

python

def grounding(t):
    report = {"citations valid": not t["cite"]["unknown"] and bool(t["cite"]["cited"])}
    weak = [c for c in t["cite"]["cited"]
            if c in BY_ID and (BY_ID[c]["status"] != "current" or BY_ID[c]["authority"] < 0.5)]
    report["cites stale or weak source"] = weak
    return report

fluent = {"answer": "Blender motors are covered for 2 years. [pol-warranty-2024]",
          "cite": cite("Blender motors are covered for 2 years. [pol-warranty-2024]", ["pol-warranty-2024"])}
print(grounding(fluent))

Code walkthrough

citations valid repeats stage 9; weak lists cited chunks archived or below authority 0.5. The scripted answer passes the first and fails the second.

Break it: hand the model only the archived terms

Now a real model, given only the archived terms.

python

stale_prompt, stale_used = assemble(Q02, [("pol-warranty-2024", 0.0)])
answer, reason, _ = generate(stale_prompt)
t = {"answer": answer, "cite": cite(answer, stale_used)}
print(answer or reason)
print(grounding(t))

Code walkthrough

One chunk, one guarded model call. Look for "2 years" citing [pol-warranty-2024]: the citation check passes, the stale check flags it. Nothing in the prompt says "archived"; only status knows.

7. The ten test questions trap (Recommended)

Teams try a first pipeline on a few questions they wrote, see good answers, and move on. Those say little about the next thousand.

The builder's ten questions score 10 of 10; the 27 labelled ones score 24, failing exactly the types the ten never asked.

Questions written while reading the documents reuse their words: here four natural plus six code, name and mixed ones. All ten pass. The 27 score 24, failing a stale, a paraphrase and an unanswerable question.

python

def passes(question, relevant):
    top = rerank(question, retrieve(question), 3)[0]
    return top[1] < 0 if not relevant else top[0] in relevant   # no answer: the best score must be weak

TEN = ["q12", "q13", "q16", "q17", "q19", "q20", "q27", "q28", "q29", "q30"]
results = {qid: passes(q, rel) for qid, q, rel, _, _ in QUESTIONS}
print("ten:", sum(results[q] for q in TEN), "of", len(TEN))
print("all:", sum(results.values()), "of", len(results),
      "| failed:", [(qid, typ) for qid, _, _, _, typ in QUESTIONS if not results[qid]])

Code walkthrough

An answerable question passes if the top chunk is right; an unanswerable one if the best score is below 0.
Prints ten: 10 of 10 and all: 24 of 27, failing q02, q07, q36.

With ten questions one question moves the score by 10 points; with 27 it moves it by 3.7 (amber: noise).

Small sets are noisy too: one question is 10 points of ten, 3.7 of 27. Grow the set from real queries, such as Kwame's log.

LinkedIn's team reached 80% of its target experience in the first month, then spent four more months trying to pass 95%.

LinkedIn's team "achieved 80% of the basic experience we were aiming to provide within the first month", then spent four more months "attempting to surpass 95%" (Bottaro and R., LinkedIn Engineering, 2024).

In production

In production every question leaves a trace per stage, which feeds debugging, cost reports and evaluation.

Teams log the whole trace, not just the answer: retrieved, reranked, prompt, reply, time and tokens per stage, so every complaint maps to a stage. LinkedIn uses small models for routing and retrieval and bigger ones for generation; its linguists evaluate up to 500 conversations a day. The Stanford authors tell users to "verify that key propositions are accurately supported by citations". Hollin gets a tracer in Module 12, an evaluation harness in Module 11.

What you added to Hollin Answers

A nine-stage pipeline in plain Python whose run returns a trace per question.
stage_checks, one cheap check per stage, and grounding, which flags stale or weak citations.
diagnose and where_did_it_go, which name the stage and the place an answer was lost.
A baseline: 24 of 27 labelled questions put a right document first.

Check yourself

Which stages run once per document, and which once per question?
The right document was retrieved at rank 2 but the answer is wrong. What do you check next?
Why does a valid citation not prove an answer is right?
"Cracked jar covered?" ranks the current terms 8th. Which of the four places is that, and for which settings?
Your pipeline passes your ten questions. What three types of question would you add first?

Parse to index on document change; retrieve to cite per question.
Whether it reached the prompt, then whether the model used it.
The source may be archived, weak or misread: misgrounded.
Missed top-k, at the prototype's top 3; at k=10 it is retrieved.
Stale versions, paraphrases, no-answer questions.

Exercise

Hannah wants current, trusted documents first. Write current_first(ranked): chunks with status "current" and authority at least 0.5 move ahead of the rest, order otherwise kept.

python

def check(current_first):
    t = run(Q02, answer=False)
    assert assemble(Q02, current_first(t["reranked"]))[1][0] == "pol-warranty-2026"
    jar = rerank("is my cracked jar covered by the warranty", retrieve("is my cracked jar covered by the warranty"))
    assert current_first(jar)[0][0] != "pol-warranty-2024", jar
    assert len(current_first(jar)) == len(jar)
    print("passed:", [i for i, _ in current_first(jar)])

# check(current_first)      # uncomment once the function is written

Code walkthrough: check confirms q02's prompt starts with the current terms, the archived terms no longer lead the cracked-jar list, and nothing is dropped.

Key terms

Term	Meaning
Indexing pipeline	Parse, chunk, embed, index: runs when documents change
Query pipeline	Retrieve, rerank, assemble, generate, cite: runs per question
Pipeline stage	One step with an output you can inspect
Trace	Each stage's output and time for one question
Silent failure	A stage gives worse output without raising an error
Retrieval, assembly, generation failure	Not found; found but not sent; sent but not used
Missed top-k	Indexed, but ranked below the cut-off
Lost in reranking	Retrieved, then pushed out by the reranker
Misgrounded	Cited, but the source is misread or does not apply
Wrong but fluent	A confident, well-written answer that is false

Sources

Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, Daniel E. Ho, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools", arXiv, 30 May 2024. https://arxiv.org/abs/2405.20362
Juan Pablo Bottaro and Karthik R., "Musings on building a Generative AI product", LinkedIn Engineering, 25 April 2024. https://www.linkedin.com/blog/engineering/generative-ai/musings-on-building-a-generative-ai-product
Nelson F. Liu and others, "Lost in the Middle: How Language Models Use Long Contexts", TACL, 2023. https://arxiv.org/abs/2307.03172
Prices: course-guide/02, from Groq and Google pricing pages (checked 9 October 2026). https://ai.google.dev/gemini-api/docs/pricing