Module 1: The problem: knowledge AI assistants cannot see

About 35 minutes · Needs: nothing; start here · Code: PLAN.md, seed/world.md, seed/access.md, seed/docs/, prototype/index.html, server/wayfinder/mcp/server.py

Where this fits

Before any server, token or index, there is an engineer with a question and an AI assistant that cannot answer it. This module looks closely at that gap at Tidewater Freight, the fictional company this course uses. Every part of Wayfinder exists to close one piece of the gap, so once the problem is clear, the rest of the design stops looking arbitrary.

What it does

Noor Rahman joined the frontend team at Tidewater Freight on 2026-10-06. In her second week she asks her assistant: "Why does billing-api retry a failed PayRail charge three times, and what happens after the third failure?" The answer exists. It is spread over eleven sources in three tools: two architecture decisions, four incident write-ups, a runbook, an older guide that says something different, a pull request, an issue that was turned down, and a README. Noor may read most of them; three of them she may not.

An AI assistant such as Claude Code, Cursor or Claude Desktop has a capable language model, but that model learnt from public text. It has never seen Tidewater's wiki, its decision records or its GitHub history. So the assistant either knows nothing about the question or, if someone connects it to everything through one shared account, knows far too much. This module explains both failures and the properties a fix needs: one place to search, a sense of what is current, a filter that follows the person asking, and citations that let a person check every claim.

Why care? A wrong answer from a confident assistant is worse than no answer. An engineer who follows a stale retry policy can double-charge customers; an engineer who is shown a restricted postmortem has learnt something they were never meant to know, and nobody can take that back.

At a glance

Use cases and examples

One index over scattered sources

• Noor's retry question needs ADR-014 from the decision log, rb-billing-retries from the wiki and billing-api#87 from GitHub. Three tools, one answer.
• A hospital's clinical guidelines live in a document system, while the reasons for a change sit in committee minutes. A nurse asking "why did the dose change?" needs both.
• A law firm keeps precedents in one system and matter notes in another. A search over only one of them misses the note that explains why a precedent was not used.

Status and freshness awareness

• guide-billing-payrail (2023) still says "5 attempts, fixed 1 s"; rb-billing-retries (2026) says 3. Quoting the older page is a real mistake, not a style issue.
• An insurer's claims handbook has three versions on the intranet. Staff must get the current one, with the old ones marked as history.
• A government office replaces a form rule each tax year. Last year's rule is still true for last year's cases, so it must stay findable but clearly dated.

Search as the asking user

• If Wayfinder searched as one service account, Noor's assistant could receive PM-004's customer figures. Searching as Noor, it cannot.
• A bank's internal assistant must not show a branch employee the board's risk minutes, even though the assistant itself could open them.
• A school's staff assistant must not show one teacher another class's safeguarding notes.

Permission-aware retrieval

• Fatima, the payments lead, sees the full PM-004 postmortem; Noor sees its public summary and a note that "the search hid 2 more matching passages".
• A SaaS company's support bot searches tickets from every customer, but each agent may only see the accounts they handle.
• A newsroom keeps embargoed stories in the same archive as published ones; an embargoed story must not even influence a search ranking.

Citations as a trust contract

• Noor's answer cites "[2] ADR-014" and labels "[3] ADR-010" as superseded background, so she can open both and see the dates.
• An online shop's policy assistant links each refund rule to the policy page, so a customer service agent can show the customer the exact paragraph.
• A government benefits helpline quotes the regulation number with every answer, so a supervisor can check it later.

A prototype first

• The prototype fixed the rule that the notice about hidden matches shows a count and group names, never titles, before any server code existed.
• A clinic's new booking screen was clicked through on paper with receptionists before the database was designed.

When not to bother. If one small team reads everything and keeps it in one well-searched tool, a plain search box and an honest "ask a colleague" may be enough. A permission filter that protects nothing only adds complexity. Citations, though, are cheap and worth having even then.

How it works

We follow Noor's question through the company's real sources, then look at the two ways a naive assistant fails, and then at what "permission-aware" must mean. Counts come from the seed (seed/world.md, seed/access.md) and the facts sheet; the answers quoted come from the live test with Claude Code on 25 Oct 2026.

Step 1: one question, eleven sources

The retry policy has a history. ADR-010 chose 5 attempts with a fixed 1 s delay in May 2023. Two incidents broke it: PM-003, a retry storm that got Tidewater rate-limited by PayRail for 41 minutes, and PM-004, where timed-out charges were retried without an idempotency key and invoices were charged twice. ADR-014 replaced ADR-010 with 3 attempts, waits of about 2 s and then 4 s, and ±20% jitter. A product request to go back to 5 attempts, billing-api#87, was closed as not planned.

Each of these sources is true for its date. None of them, read alone, tells you which one is current.

Step 2: knowledge lives in many places

Tidewater Freight has 403 sources in six kinds: 87 docs in the wiki, 25 architecture decision records, 153 issues and 132 pull requests across three repositories (billing-api, routing-service, web-portal), plus a README and a CODEOWNERS file for each repository. Split into searchable pieces, they make 1,154 chunks.

Most of the history is in GitHub: a pull request says what changed, and its review comments say why.

Step 3: two pages disagree

Stale documentation is a page that was correct when written and was never updated. guide-billing-payrail, last updated on 2023-09-04, still says "5 attempts in total" and "fixed 1 s delay between attempts". The canonical runbook rb-billing-retries, rewritten on 2026-03-10, says 3 attempts with 2 s and 4 s waits. Both are still in the wiki. The seed calls this trap T4: a good answer comes from the newer page and, ideally, mentions the stale one.

Step 4: decisions change, and rejected ideas stay around

An architecture decision record (ADR) is a short page that records one decision, its context and its consequences. When a later decision replaces it, the old ADR is not deleted: its status becomes superseded and it points to its replacement. The same is true for issues. billing-api#87 asked to raise the attempts to 5; it was closed with the reason not_planned, which means rejected, not done. An assistant that reads "raise attempts to 5" without its state would describe a plan that was turned down.

Step 5: not everyone may read everything

Each doc carries a visibility label. Of the 87 docs, 49 are public, 30 are readable only by one team (plus leadership), and 8 are restricted to a small group: five to the security team and three, including the full PM-003 and PM-004 postmortems, to payments and security (plus leadership). ADRs, issues and pull requests are all public, and the seed rules forbid them to contain restricted detail. Module 7 (access model) explains how a label becomes a set of groups per person.

Step 6: failure 1, the assistant sees nothing

Without a connection to Tidewater's sources, the assistant's model can only answer from what it learnt in training. It can explain retries and backoff in general. It cannot know that ADR-014 replaced ADR-010, that billing-api#87 was turned down, or what the settlement worker does after the third failure. Worse, a plausible guess looks exactly like a fact when nothing is cited.

Step 7: failure 2, the assistant sees too much

The quick fix is to load every source into one search index and let the assistant query it through a service account, one technical account with access to everything. Now the assistant finds the right sources. It also finds the wrong ones. When Noor asks "What happened in the duplicate-charge incident?", the best-matching passage is in pm-004, the restricted postmortem, with 312 invoices, 58 shipper accounts and USD 187,400. The index, not the reader, decided what was readable. Wayfinder never does this; the figure shows what a shared account would return.

Step 8: the same question, two people

What we want instead is a search that runs as the person asking. In the live test on 25 Oct 2026 (Claude Code 2.1.302, with personal access tokens for Noor and Fatima that were revoked afterwards), both asked about the duplicate-charge incident. Noor's answer came from pm-004-summary and billing-api#39. It said the full postmortem is restricted to the payments team, that "the search hid 2 more matching passages", and suggested asking #team-payments. Fatima's answer cited pm-004 and gave 312 invoices, 58 shipper accounts and USD 187,400 refunded, and it ended: "Before you share this: pm-004 is restricted to payments-incidents ... cite pm-004-summary [1] instead".

Both answers are correct, for different readers.

Step 9: every team reads a different set

The seed's access rules give each team a different view: leadership reads 82 of the 87 docs, payments 60, platform and security 59 each, routing 55 and frontend 53. Leadership is not all-seeing: it does not read the five restricted:security docs. So "filter by role" or "admins see everything" would both be wrong here; the filter must compute each person's view.

Step 10: what permission-aware has to mean

Permission-aware retrieval is search that only ever considers what the asking person may read. Three rules make it real. The filter runs inside the database query, before ranking, so a restricted passage is never scored or fetched. A source the caller may not read is answered as "not found", exactly like a source that does not exist, so nobody can probe ids. And hidden matches are only counted: the result says how many sources were withheld and from which groups, never their titles. Module 10 (permission filtering) shows the SQL and the tests.

Step 11: citations make answers checkable

A citation is a pointer from a claim to its source. In Wayfinder each one carries five fields: source_id, title, kind, url and updated. The server has no language model; the assistant's model writes the sentences. Citations are the contract between the two: the model may only state what a cited source says, and the reader can open the source, check its date and see whether it is current. In Noor's retry answer, ADR-010 appears as "[3] ... superseded by ADR-014; background only".

Step 12: a prototype before the build

Before any server code existed, the author built prototype/index.html: a single 1,051-line page with the console's seven screens (Overview, Connect your client, Search playground, Tools and resources, Audit log, Access matrix, Index and evaluation), six personas and scripted search results. Signed in as Noor, the duplicate-charge search shows a notice: "Filtered out for you: 3 restricted passages from restricted:payments-incidents. They were excluded in SQL before ranking, so they were never scored and their titles are not shown to you or your AI client." The build plan copied that rule word for word: the "filtered out" notice shows a count and group names, never titles.

Code tour

This module is mostly about evidence: the seed company and the plan. Seed files are shown as Markdown, the issue as plain text, and one short excerpt from source-code/ shows where the citation rule lives in the product.

New concept: front matter Front matter is a block of key: value lines between two --- lines at the top of a Markdown file. It holds facts about the page (id, kind, team, visibility, owner, last update) separately from its text. Wayfinder's loader reads these fields into columns, which is how it knows a page's visibility and date without guessing from the prose.

The access rules are a table in the seed, written for people first:

<!-- seed/access.md -->
## Visibility values

| Visibility | Who can read |
|---|---|
| `public` | everyone (all 41 people) |
| `team:<id>` | members of that team, plus `leadership` |
| `restricted:payments-incidents` | payments + security, plus `leadership` |
| `restricted:security` | security only (leadership does **not** see it) |

## Groups per team

A person's groups are derived from `people.json` `team`. Being a lead or staff engineer grants nothing extra.

| Team | Members | Groups the member holds | Visible docs (of 87) |
|---|---|---|---|
| platform | 9 | `public`, `team:platform` | 59 |
| payments | 8 | `public`, `team:payments`, `restricted:payments-incidents` | 60 |
| routing | 8 | `public`, `team:routing` | 55 |
| frontend | 8 | `public`, `team:frontend` | 53 |
| security | 4 | `public`, `team:security`, `restricted:security`, `restricted:payments-incidents` | 59 |
| leadership | 4 | `public`, every `team:*`, `restricted:payments-incidents` | 82 |

Code walkthrough

• The first table is the meaning of each label. team:<id> stands for six labels, one per team.
• restricted:security is the only label leadership does not hold, which is why Priya, the VP of Engineering, cannot read PM-006 or PM-009.
• "Being a lead or staff engineer grants nothing extra" is a deliberate rule: Fatima reads pm-004 because she is in payments, not because she is a lead.
• The last column is a test oracle: test_visible_doc_counts_per_team checks the code against these numbers.

Here is the stale guide's retry section:

<!-- seed/docs/guide-billing-payrail.md -->
---
id: guide-billing-payrail
title: PayRail integration guide for billing-api
kind: guide
team: payments
visibility: team:payments
owner: p-arjun-kapoor
updated: 2023-09-04
links: [ADR-010, ADR-011, ADR-004, ADR-012, pm-003-summary]
---
<!-- ... -->
## Retry policy

PayRail has occasional short brownouts. The client retries failed calls following ADR-010:

- **5 attempts in total** (1 initial + 4 retries),
- **fixed 1 s delay** between attempts,

Code walkthrough

• visibility: team:payments means Noor cannot read this page at all; Fatima can, so she is the one who might be misled by it.
• updated: 2023-09-04 is the date that lets a search prefer the newer runbook. Module 9 (ranking) shows the age penalty for guides and runbooks.
• links lists ADR-010, the decision the guide follows. Nothing in the guide says ADR-010 was superseded two months later; the page simply stopped being maintained.

And the canonical runbook that replaced it:

<!-- seed/docs/rb-billing-retries.md -->
---
id: rb-billing-retries
title: "Runbook: billing-api PayRail retries and payment_pending_retry"
kind: runbook
team: payments
visibility: public
owner: p-ben-okafor
updated: 2026-03-10
links: [ADR-014, ADR-013, ADR-010, ADR-023, pm-003-summary, pm-004-summary, arch-003, guide-idempotency, hb-oncall, guide-deploy, pol-001]
---

# Runbook: billing-api PayRail retries

This is the **canonical** description of how `billing-api` retries charges against PayRail. It was rewritten on
2026-03-10 and replaces every older description. If another page (notably the 2023 PayRail integration guide in the payments
team space) says "5 retries with a fixed 1 s delay", that page is stale: that was ADR-010, superseded by
ADR-014 on 2023-11-20.

## The policy (ADR-014)

| Setting | Value |
|---|---|
| Attempts | **3 in total** (1 initial + 2 retries) |

Code walkthrough

• visibility: public: every engineer, Noor included, may read the current policy.
• Its links point only at public sources (the two incident summaries, not the restricted postmortems). This is the link-visibility rule at work.
• The text names the stale guide by description ("the 2023 PayRail integration guide in the payments team space"), not by id. That wording is a fix from the seed audit (see What can go wrong).

New concept: superseded A decision is superseded when a later decision replaces it. The old record stays, with status: superseded and a superseded_by field naming its replacement, so the history can still be read. A search that returns a superseded ADR must also return what replaced it, or the reader gets yesterday's rule.

<!-- seed/adrs/ADR-010.md -->
---
id: ADR-010
title: "billing-api PayRail retries: 5 attempts, fixed 1 s delay"
status: superseded
date: 2023-05-15
deciders:
  - p-fatima-alsayed
superseded_by: ADR-014
service: billing-api
visibility: public
---

Code walkthrough

• status: superseded and superseded_by: ADR-014 are the two fields Wayfinder reads to attach ADR-014 whenever ADR-010 is found.
• deciders uses person ids, as every seed file does, so ownership questions can be answered from data.
• ADRs are all public, so the history of a decision is never hidden, only its restricted evidence.

The rejected request is one line of seed/repos/billing-api/issues.jsonl (shortened here; the comments follow):

{"number": 87, "title": "Raise PayRail retry attempts to 5 to cut failed payments", "body": "Product request. We still see invoices end up in `payment_pending_retry` after PayRail hiccups. Raising the in-request attempts from 3 to 5 should turn more of those into immediate successes and fewer follow-up emails.", "state": "closed", "state_reason": "not_planned", "labels": ["retries", "payrail", "wontfix"], "author": "p-tomas-lindqvist", "assignees": ["p-fatima-alsayed"], "created": "2025-01-21", "closed": "2025-02-11", ...}

Read without "state_reason": "not_planned", this issue sounds like a plan. With it, it is evidence that the team considered 5 attempts and said no.

New concept: citation contract A citation contract is an agreement between the tool that finds sources and the model that writes the answer. The tool returns each passage with a numbered citation; the model promises to cite every claim and to state nothing that no cited source says. The contract cannot be enforced by the server, so it is written down as instructions the client receives when it connects.

# server/wayfinder/mcp/server.py
INSTRUCTIONS = """\
Wayfinder is Tidewater Freight's engineering knowledge hub: docs, runbooks, postmortems, architecture decisions
(ADRs), GitHub issues and pull requests. It has no language model; you write the answer from what it returns.
# ...
- Every result lists numbered citations {source_id, title, kind, url, updated}. Cite every claim with [n] and the
  source_id (e.g. "[2] ADR-014"), and include the url when the user may want to open it. Do not state facts that
  are not in a cited source; say what is missing instead.
# ...
def _not_found_text(ref: str) -> str:
    # Identical for "does not exist" and "exists but you may not read it" (never reveal which).
    return f"Not found: {ref}. It does not exist or you do not have access to it."

Code walkthrough

• INSTRUCTIONS is sent to every client when it connects (Module 3, MCP instructions). The first lines say plainly that the server has no language model.
• The citation bullet names the five citation fields and asks for "[n]" plus the source_id on every claim, the exact shape of Noor's answers.
• "say what is missing instead" is why Noor's answer has a Gaps section rather than a guess.
• _not_found_text is the "not found means not found" rule: one sentence for a missing id and for an id the caller may not read.

What happens when Noor asks about the duplicate-charge incident

This trace follows the live test (q3, 25 Oct 2026) through the code; Modules 2 and 10 cover the solution and the permission filter step by step.

1. Claude Code, run as claude -p with Noor's personal access token, decides to call search_knowledge with a query its model wrote (the exact query is not recorded).
2. Wayfinder resolves the token to Noor: user p-noor-rahman, team frontend (mcp/principal.py).
3. Her groups are public and team:frontend (access.py, from seed/access.md).
4. Every ranked SQL query includes c.visibility = ANY(:groups), so chunks of pm-004 and rb-billing-duplicate-charge (restricted:payments-incidents) are never scored.
5. Readable passages come back with numbered citations, among them pm-004-summary and billing-api#39.
6. The same ranking runs once more over ids and visibility labels only, and counts the sources she may not read that would have made her list: the withheld count. Its value is not in the transcript; the answer reports "the search hid 2 more matching passages".
7. Claude Code's model writes the answer from the summary, cites [1] pm-004-summary and [3] billing-api#39, states that the full postmortem is restricted, and suggests asking #team-payments or the owner, p-fatima-alsayed.
8. The session took 4 turns and 16,624 ms. The cost, USD 0.175, was Claude Code's own model usage; Wayfinder has no model and no per-call model cost.

Design decisions

The decision that shapes everything later is where the filter runs. Ranking first and hiding afterwards seems simpler, but it returns fewer than eight results when restricted passages ranked high, and the gaps and scores hint at what was removed. Filtering inside the query costs nothing extra at this size: the permission suite's searches took 11.9 ms at p50 (run 20261025T113119Z-permissions).

When to choose differently

What can go wrong

Every trap in the seed company is a failure an assistant could make. Each has a defence and a test.

A stale page wins. A search that ignores dates could rank the 2023 guide above the 2026 runbook. Wayfinder gives how-to style docs a gentle age penalty and the retrieval eval checks the trap questions (trap_newer_first 1.0 on 9 checks). Test: test_newer_doc_ranks_above_stale_one.

A superseded decision is presented as current. If ADR-010 is found, ADR-014 is attached to it and listed right above it. Test: test_superseded_adr_comes_with_replacement_above_it.

A rejected issue reads like a plan. Issues carry their state and state_reason, and results show not_planned. Test: test_not_planned_issue_is_labelled.

A restricted passage reaches an outsider. The access filter sits in the WHERE clause of every ranked query, and the withheld count is computed from ids and labels only. Test: test_restricted_postmortem_hidden_but_counted.

Fixed: a public runbook named a team-only page.

The seed audit on 25 Oct 2026 found that rb-billing-retries, a public runbook, listed guide-billing-payrail (team:payments) in its links. No secret text leaked, but every engineer could learn the guide's id and that it exists. The audit adopted the link-visibility rule: an item may reference another only if every reader of the first can read the second, and in-text id mentions count. About 45 references were fixed this way (the changelog lists 43 in 24 items), by removing the link and describing the owning team's page instead. The seed validator now enforces the rule on every run, and in CI, and the indexer applies the same rule when it writes chunk text. Module 6 (the seed company) tells the whole audit, including the 49 contradictions it resolved.

Fixed: a doc cited a pull request that did not exist yet. pm-004, dated 2023-10-12, cited billing-api#61, which was created on 2023-10-18. A careful reader would distrust the whole document. The audit found this through the validator's date checks and reworded it to "billing-api PR in progress"; ten anachronisms were fixed in total.

Fixed: cross-references could name restricted sources. The security review found that a public source whose links or superseded_by named a restricted doc would reveal that id in get_document. Now unreadable cross-reference ids are hidden, and search attaches a replacement only if the reader may read it. Tests: test_get_source_does_not_reveal_unreadable_cross_references, test_search_never_attaches_an_unreadable_replacement.

Known limits. The withheld count tells a user that something matched, with the group names, though never what. The citation contract is an instruction to the client's model; the server cannot force a model to cite, or stop it from writing something no source says. Issues and pull requests are public in the seed; a real company with private repositories would need their visibility too. And Tidewater Freight is fictional: its 403 sources are clean and consistent in a way real wikis rarely are.

Try it yourself

Start the local stack (needs Docker):

cd source-code
docker compose up -d --build

What you should see

Three containers: postgres, server and console. The first start migrates the database, loads the company and builds the index, which takes about a minute. The console is then at http://localhost:8096.

In the console, sign in as noor.rahman@tidewater.example with the demo password Tidewater-Demo-2026 (local demo only). Open Search, ask "What happened in the duplicate-charge incident?", and read the notice above the results. Sign out, sign in as fatima.alsayed@tidewater.example and ask again: pm-004 appears, and the notice is gone.

Look at the seed itself, from the project root:

grep -l "^visibility: restricted" seed/docs/*.md
uv run --with pyyaml python seed/tools/validate.py

What you should see

Eight files (pm-003, pm-004, pm-006, pm-009, rb-billing-duplicate-charge, rb-secret-rotation, sec-003, sec-005), then 0 errors, 103 warnings (run with --warnings to list warnings). Open seed/docs/guide-billing-payrail.md and seed/docs/rb-billing-retries.md side by side and find the two retry policies.

Check yourself

1. Why is "load everything into one index and query it with a service account" not a fix?

The service account can read everything, so the index decides what is readable, not the person asking. Noor's question about the duplicate-charge incident would return passages from the restricted pm-004, with its customer figures.

1. ADR-010 and ADR-014 are both public and both accurate for their dates. What must a search do when it finds ADR-010?

Return ADR-014 with it and present ADR-014 as the current decision, ADR-010 only as history. Wayfinder attaches the replacement and lists it right above the superseded ADR.

1. Noor asks for pm-004 by id. What should she get, and why exactly that?

"Not found: pm-004. It does not exist or you do not have access to it." The same text as for an id that does not exist, so she cannot use the answer to learn which restricted ids exist.

1. Why does the hidden-passage notice show a count and group names but never titles?

The count and the group tell the reader that more exists and whom to ask. A title such as "PM-004: duplicate charges on 312 invoices" would itself leak restricted facts.

1. Fatima can read the stale guide; Noor cannot. Who is more at risk from it, and what protects them?

Fatima, because only payments, and leadership, can see it. The newer runbook ranks higher through the age penalty for how-to docs, and the runbook itself says the older page is stale.

Key terms