CourseLarge Language Models: The Complete Course · Course guide · part 4 of 103
Part 4 · Course guide

The Project: the Quillpath Support Desk

14 min read·13 Oct 2026

Every lesson in this course is set at one company, on one project. Quillpath is made up. All people, workspaces, tickets, invoices, rules, dates and numbers in the project are fictional and simulated. The real cases that open each lesson are separate, cited, and clearly marked as real.

The company

Quillpath sells project-management software online: boards, cards, checklists and automations, used by small teams around the world. It has four plans, Free, Team, Business and Enterprise, priced per member seat. Customers write to support in English, German, Spanish, Japanese and Hindi.

Money in this course is in US dollars, the currency Quillpath bills in and the currency the model providers price in.

The problem

Quillpath's support desk receives about 1,800 tickets a week and has six agents. The median ticket waits 7.5 hours for a first reply. Most tickets are the same few questions: a charge that looks wrong, a locked account, how to export data, whether a refund is due. The answers are already in the help centre.

Maya, the Support Lead, wants an assistant that:

  1. Triages every ticket: a category, a priority, and the help-centre article that answers it, if there is one.
  2. Drafts a reply from the help centre, citing the article, for an agent to review before it is sent.
  3. Says "this needs a person" when no article answers the ticket.
  4. Handles refund requests by the written policy, and never moves money without a person approving.
  5. Costs at most one cent of model fees per ticket, on average.

You are the engineer building it.

What this course uses the project for

This course is about the model itself: what it is, how it reads tokens, how it chooses words, how to prompt it, how to make it reliable, how to adapt it, how to test it, how to secure it and how to run it. The support desk is the instrument. Every idea is measured on the same tickets, so you see what a technique changes on a task you already know.

It is not the BrightCart store of the AI Engineering course, the Harbourline Freight engagement of the Forward Deployed Engineer course, or the returns desk of the Agentic AI Engineer course. Quillpath is a different company, with a different product, policy and cast.

The people

NameRoleIn the project
MayaSupport LeadOwns the queue and the refund policy. Brings the problems. Decides what the policy means when it is unclear. Approves refunds up to 500 USD
JoelBilling Operations LeadOwns the model budget and sets the one-cent cap. Approves refunds above 500 USD
AnikaSecurity EngineerAsks how the assistant can be abused. Runs the red-team session in week 8
TomaszPlatform EngineerOwns the billing system the tools call, and the deployment. Proposes the model switch in week 13
Lena and RaviSupport agentsLabel the tickets from written definitions, review drafted replies, and act as the two human raters in Module 10
YouAI engineerBuild the assistant, from the first model call to the deployed service

Customers who recur in examples: Ana Duarte, owner of Ardent Design (workspace W-201), who sends T-1001 about a duplicate charge; Leo Park, a member of the same workspace, who sends T-1034; Grace Mwangi (Hollis Bakery, W-202), Jonas Feldmann (Feldmann Architekten, W-203), Yuki Sato (Studio Nami, W-204) and Ruth Pell (Pell & Rowan, W-205 and W-206). Minor characters may appear in a single lesson. The main cast above does not change.

The rules

RuleValue
PlansFree; Team 12 USD per member per month or 10 USD billed annually; Business 24 USD or 20 USD annually; Enterprise custom
RefundsMonthly plans are not refunded pro rata. Annual plans cancelled within 14 days of purchase or renewal get a full refund (day 14 is inside); after 14 days, none. Duplicate charges are always refunded in full
Who may askOnly workspace owners and billing admins may cancel or request a refund
ApprovalA refund is issued only after a person approves it: Maya up to 500 USD, Joel above 500 USD
Sign-in5 failed attempts lock an account for 15 minutes; password reset links expire after 30 minutes
SSOBusiness and Enterprise only
DeletionA deleted workspace can be restored by its owner for 30 days
Support hoursMonday to Friday, 07:00 to 19:00 UTC; urgent tickets on Business and Enterprise get a first reply within 1 hour at any time
Model spendAt most 0.01 USD of model fees per ticket, on average

Code enforces every rule that moves money or touches an account. The prompt only describes them. The exact wording, the help centre, the 36 labelled tickets, the billing records and the refund rule as code are in 06-shared-data.md.

Quillpath's systems

SystemOwnerWhat the assistant may do with it
Ticket queueMayaRead a ticket; write a triage label and a draft reply for an agent
Help centreMayaSearch and read the twelve published articles
Billing systemTomaszRead workspaces and invoices; request a refund, which waits for Maya or Joel
Status pageTomaszRead current incidents

From Module 6 on, the assistant reaches these systems through five tools with fixed names, so that every module calls them the same way:

ToolKindWhat it does
search_help_centre(query)ReadReturns the best-matching article ids and titles
get_workspace(workspace_id)ReadReturns plan, billing period and seats
get_invoice(invoice_id)ReadReturns date, amount and the charges taken
issue_refund(invoice_id, amount_usd, reason)Moves moneyChecks decide and the amount in code, then waits for a person to approve
escalate(ticket_id, reason)Hands overPuts the ticket in a person's queue and stops

The structured outputs also have fixed names. Triage holds category, priority, kb_article (an article id or None) and needs_human. DraftReply holds text, citations (article ids) and needs_human. Lessons may add fields; they do not rename these. Module 6 Lesson 1 adds one optional field to Triage: amount_usd, the amount the ticket names, from 0 to 10,000 USD, or None.

TinyLM and the other models

Hosted models do not let you look inside, so the course trains its own: TinyLM, a GPT-style model with 1,071,872 parameters (a 2,048-slot vocabulary for byte pair encoding tokens (the tokeniser trained on the shared corpus fills 1,134 of them, because every word is already one token), context 128 tokens, width 128, 4 layers, 4 attention heads, tied input and output embeddings). Module 1 builds and trains it in Colab on the corpus from 06-shared-data.md, and later modules open it up to show probabilities, sampling, the KV cache, attention dilution and LoRA. It is useless as an assistant and excellent as an instrument. Never read its output quality as evidence about real models.

The hosted models are the course defaults in 02-setup-colab-and-llm-providers.md. Where a lesson needs a small open model it can download and adapt, it uses SmolLM2-135M-Instruct from Hugging Face (Apache 2.0, checked 9 October 2026).

The canonical offline token counter is tiktoken's o200k_harmony encoding, which tiktoken maps to the gpt-oss models (checked in tiktoken 0.14.0). Provider counts include chat formatting and can differ by a few tokens; Module 2 shows how much.

Dates

  • Course date: Friday 9 October 2026. Model names, prices and documentation were checked on this day. Lessons say "(checked October 2026)".
  • The project: Monday 1 June 2026 to Friday 11 September 2026, fifteen weeks. Lessons are told in the present tense, as if you are there.
  • The data date: Monday 8 June 2026, in week 2. The 36 tickets are a snapshot of that morning's queue. Code measures "days since" from that date (DATA_AS_OF), never from the computer's clock.

Key events

Lessons refer to these events. They are fixed, so that every module tells the same story. Week N starts on the Monday shown. All of them are simulated.

WhenWeekEvent
Monday 1 June 20261Maya asks for the assistant. Her team measures the baseline: 1,800 tickets a week, a median first reply of 7.5 hours. You choose candidate models with a written selection framework
Monday 8 June2The 36 tickets are taken from that morning's queue. Lena and Ravi label them from the written definitions. The dev and test split is frozen and never changes
Wednesday 17 June3Joel sets the spend cap: one cent of model fees per ticket. You count tokens and price triage and drafting before sending anything
Monday 22 June4Triage prompt v1 is committed as a file, with a control prompt, and the harness runs on the dev set
Monday 29 June5You and Maya write the refund rule as code. She decides that day 14 is inside the window
Monday 6 July6Triage returns structured output. The five tools exist; issue_refund waits for a person
Monday 13 July7The assistant drafts replies from retrieved articles, with a context built to a token budget, and remembers what customers have told it
Tuesday 14 July7Maya approves Ana's duplicate-charge refund (T-1001). A fixed fact: Module 7 Lesson 4 uses it to show that a status copied into memory on 13 July is stale by 14 July, while the billing system has the answer
Monday 20 July8The refund agent runs on the money tickets, traced step by step. In Anika's red-team session, T-1034 tells the assistant to refund 500 USD without asking. The role check in code refuses it
Monday 27 July9A fine-tuning trial on a small open model, against the prompted baseline. The decision and its numbers are written down
Monday 3 August10Shadow mode: every draft is reviewed by Lena or Ravi before it is sent. Their ratings calibrate an LLM judge
Tuesday 11 August11The pilot incident. Ravi approves a German draft promising a full refund to an admin, 30 days after renewal. Joel refuses the refund at the approval step, so no money moves, but the customer was promised it. Maya apologises the next day
Monday 17 August12Customers start attaching screenshots and invoice PDFs, and some send voice messages from the mobile app
Monday 24 August13Tomasz proposes moving triage to a stronger model on quality grounds (on Groq, openai/gpt-oss-120b, which costs more per token than the default). You run the regression suite before switching, and put a gateway with a fallback in front of both providers
Monday 31 August14Limited live: drafted replies for billing and sign-in tickets, behind a feature flag, with a written rollback plan
Friday 11 September15Review with Maya and Joel: your write-up, with numbers from your own runs, and a memo on what the assistant must not be used for

The course does not fix the assistant's accuracy, cost or speed in advance. Those numbers come from running your own code, and they are what your write-up reports.

What you build, module by module

ModuleWhenWhat the support desk gains
1. What a Large Language Model IsWeeks 1 to 2The shared data, one client for three providers, a first triage call, TinyLM, a model selection
2. Tokens, Context, and CostWeek 3One token counter, one price table, a context budget and a cost per ticket
3. Inference and DecodingWeek 3Decoding settings chosen per task, streaming, and a measured view of reasoning cost
4. Prompt EngineeringWeek 4Versioned triage prompts, a frozen test set and a harness that scores them
5. Reasoning and Advanced PromptingWeek 5Refund decisions by extract-then-rule, voting and a verifier
6. Structured Outputs and Tool UseWeek 6Typed triage output, repair loops, the five tools and their guards
7. Context EngineeringWeek 7Help-centre retrieval, a context builder with section budgets, and memory
8. AgentsWeek 8A traced refund agent with budgets and an approval gate, also served over MCP
9. Fine-Tuning and AdaptationWeek 9A LoRA trial on a small open model and a written decision
10. EvaluationWeeks 10 to 11An eval suite with confidence intervals, human ratings, a calibrated judge and a CI gate
11. Safety, Security, and AlignmentWeeks 8 to 11Measured defences against injection, redaction of personal data, a red-team set
12. Multimodal ModelsWeek 12Intake for screenshots, PDFs and voice messages, scored field by field
13. Deployment, Operations, and EconomicsWeeks 13 to 14A gateway with fallback, caching and quotas, a web service, tracing and a rollback plan
14. Applications, Patterns, and FrontiersWeeks 14 to 15Review placement, feedback capture and a limitations memo
15. CapstoneWeek 15One of four capstone tracks, with shared evidence requirements

One repository: which module adds which file

Everything you keep goes in one repository, a single Python package called supportdesk/, in a folder called supportdesk-repo (on your Google Drive in Colab, so it survives between sessions). Lessons run in Colab, so their code sits in cells; the labs write each piece to its place in the repository. Every lab from Module 2 on starts by checking that the earlier modules' files are there and writing back any that are missing, so you can start at any module. This table lists what each module's lab writes, checked against the labs on 13 October 2026.

ModuleAdds to the repository
1The package itself: supportdesk/__init__.py, supportdesk/data.py (the shared data), supportdesk/llm.py (the setup cell, llm created once, and ask), supportdesk/baseline.py (the first triage call), supportdesk/tinylm.py; tests/conftest.py, tests/test_data.py, tests/test_baseline.py, tests/test_tinylm.py; pyproject.toml; .gitignore. TinyLM's trained weights go in models/ (tinylm.pt, tinylm-tokenizer.json), which is git-ignored and rebuilt in about a minute
2supportdesk/tokens.py (the one counter), supportdesk/pricing.py (PRICES from course-guide/02, with cached and batch rates, and cost_usd); tests/test_tokens.py, tests/test_pricing.py
3supportdesk/decoding.py (the sampling filters and generate for TinyLM, decoding settings per task, stream_timed and timings, the finish check finish_reason and checked, and the reasoning-effort helpers thinker and thinking_tokens); tests/test_decoding.py
4prompts/control.md, prompts/triage_v1.md (and prompts/boundary.md, written once in the lab's tuning step), supportdesk/prompts.py (load_prompt, render, parse_triage), supportdesk/harness.py; runs/ (run records and runs/split_manifest.json); tests/test_harness.py
5supportdesk/refunds.py (the extract-then-rule chain, voting and the verifier; decide itself stays in supportdesk/data.py); tests/test_refunds.py
6supportdesk/schemas.py (Triage, DraftReply), supportdesk/tools.py, supportdesk/guards.py; tests/test_schemas.py, tests/test_guards.py
7supportdesk/kb_search.py, supportdesk/context.py, supportdesk/memory.py; tests/test_kb_search.py, tests/test_context.py, tests/test_memory.py
8supportdesk/approvals.py (the refund queue), supportdesk/trace.py, supportdesk/agent.py (one run_agent, budgets and checkpoints), supportdesk/trajectory.py, supportdesk/mcp_server.py; state/ for run state (refund_queue.json, traces.jsonl, checkpoints/), git-ignored by its own state/.gitignore; tests/test_agent.py
9finetune/: __init__.py, plan.py, smol.py, baselines.py, build_data.py, train_lora.py, prefs.py, decision.py; the data files eval_plan.json, sft_train.jsonl, preference_pairs.jsonl; finetune/results/ (baselines.json, holdout.json, triage_lora.json, reply_dpo_decision.json, decision.md); finetune/adapters/, git-ignored by finetune/.gitignore; tests/test_finetune.py
10evals/: __init__.py, checks.py, golden.py, run.py, stats.py, rubric.py, agreement.py, judge.py, judge_calibration.py, gate.py, gate.json; the frozen data cases.jsonl, golden.json, regressions.jsonl, and baseline.jsonl; .github/workflows/eval-gate.yml (the eval gate in CI); tests/test_evals.py
11supportdesk/redact.py, supportdesk/security.py, evals/redteam.jsonl; tests/test_security.py
12supportdesk/intake.py (images, PDFs and voice; check_record); tests/attachments.py (makes the test invoices and audio), tests/test_intake.py; samples/ for the generated test files, git-ignored by its own samples/.gitignore (written by the lab)
13supportdesk/gateway.py (fallback, caching, quotas), supportdesk/otel_trace.py (OpenTelemetry tracing beside Module 8's trace.py), supportdesk/api.py, supportdesk/releases.py; docs/rollback.md; requirements-service.txt, Dockerfile, .github/workflows/ci.yml (tests and the container build, beside Module 10's eval-gate.yml); tests/test_gateway.py
14supportdesk/review.py (the send gate and review queue), supportdesk/feedback.py (the reviews table); docs/limitations.md; tests/test_review.py. Reviews are stored in runs/reviews.db, a SQLite file that is never committed: the lab writes runs/.gitignore to keep it out of Git
15supportdesk/tracks.py (the four capstone tracks), evals/evidence.py; docs/rules.json, docs/evidence.json, docs/write-up.md; runs/capstone-<track>-test.jsonl and runs/capstone-<track>-redteam.jsonl; README.md; tests/test_capstone.py

Two things are not in any package file. The pilot incident (PILOT_INCIDENT, from 06-shared-data.md) is a notebook constant: lessons and labs that need it paste it from the shared data, and no module adds it to supportdesk/. Module 1's lab says Module 10 adds it; it does not. Module 8's retry with backoff also stays in lesson cells.

The finished support desk

  • Triage of every ticket into typed, validated output
  • Grounded draft replies with citations, reviewed by a person
  • Refund decisions made by a rule in code, with a person approving every refund
  • Tools with argument checks, least privilege and errors the model can act on
  • A context builder with token budgets, and memory with provenance
  • A measured cost per ticket under the one-cent cap
  • An eval suite with confidence intervals, human ratings and a calibrated judge, gating every change
  • Defences against prompt injection, measured, and redaction of personal data
  • A gateway with fallback across providers, caching, quotas and tracing
  • A web service in a container, with a rollback plan
  • A write-up and a limitations memo with numbers from your own runs

The one idea

A model that is right 90% of the time is not a product. The product is what you build around it: the test set that tells you the 90%, the rule in code that catches the 10% that moves money, and the person who sees what the code cannot check. The pilot incident in week 11 is what happens when the last of those is overloaded.

Changes from the earlier version (for authors)

The earlier version of this course set the same project at a company called Brightlane. What was kept and what changed:

  • Renamed. Brightlane became Quillpath, because "Brightlane" was too close to BrightCart, the store in the AI Engineering course, and both projects build a support assistant. The support lead, Maya, the twelve help-centre article ids, the plans and prices, the refund policy and the supportdesk/ package name are kept.
  • Smaller data. 72 tickets became 36 (24 dev, 12 test), so a lesson can paste the slice it needs. The six categories, four priorities, gold article and answerable flag are kept, and the categories now have written definitions.
  • TinyLM kept, built in the course. It was a shipped file; the platform has no downloads, so Module 1 now trains it in Colab from the corpus in 06-shared-data.
  • Real outputs only. The earlier version showed some hand-written stand-in outputs, labelled illustrative. Every output in this course comes from a real run. A scripted stand-in may still be used to test plumbing (parsers, loops, retries), and is then named as such and never presented as a model's answer.
  • Prices fixed. The earlier price table billed Groq's cached input at the full input price. Groq gives cached input a 50% discount; the dated rates are in course-guide/02.
  • Tokeniser dropped. An unsourced "claude-legacy" tokeniser was removed. Anthropic's token counting appears only as an official-docs snippet, labelled "not run here".
  • New cast. Joel, Anika, Tomasz, Lena and Ravi are named owners for the budget, security, the platform and the labels. The pilot incident is new.