Lesson 5: Choosing a model when the landscape keeps moving

Module 1 · Lesson 5 · About 45 minutes · Needs: Lesson 4 (scoring a model on the 24 dev tickets, the Wilson interval)

The situation

It is Tuesday 9 June 2026, week 2. In week 1 you promised Maya a written model selection. Now everyone has an opinion. Tomasz wants a small open model on Quillpath's own servers. Joel repeats his cap: one cent of model fees per ticket. A colleague sends a leaderboard and says "take the top one".

This lesson writes it, tracing T-1004 again: Yuki Sato wants her money back on an annual plan bought 20 days ago, gold label cancellation.

A real case, first

In April 2025 Meta released Llama 4 Maverick, and it appeared near the top of LM Arena, a public leaderboard where people vote between two anonymous chat replies. Kyle Wiggers reported in TechCrunch ("Meta's vanilla Maverick AI model ranks below rivals on a popular chat benchmark", 11 April 2025) that the arena entry was an experimental, unreleased version that Meta said was "optimized for conversationality". LM Arena's maintainers apologised, changed their policies and scored the released model, "Llama-4-Maverick-17B-128E-Instruct". It ranked below GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro.

Later that month, Singh and twelve colleagues published "The Leaderboard Illusion" (arXiv, 29 April 2025). They identified "27 private LLM variants tested by Meta in the lead-up to the Llama-4 release", and estimated that Google and OpenAI had received 19.2% and 20.4% of the arena's data, against 29.7% for 83 open-weight models together. The model you can download is not always the model on the leaderboard.

What you will be able to do

• Place a model in a tier and price one ticket on each tier.
• Tell a proprietary API from an open-weight model, and say what each costs you.
• Measure the hidden tokens a reasoning model spends.
• Read a model card's benchmark table and say what it does not tell you.
• Select a model with a written rule that uses your own measurements.

1. Frontier, mid-tier and small models (Essential)

Providers sell the same idea at very different prices. Picking the wrong tier can cost Quillpath over a hundred times more for the same label on T-1004.

Every large provider sells a family in three sizes. A frontier model is the provider's most capable and most expensive. A mid-tier model trades a little quality for speed and price. A small model is built for high volume: classification, extraction, routing. Anthropic's model page, for example, describes Claude Haiku 5.5 as "for high-volume, latency-sensitive tasks such as classification, extraction, and routing" (checked 12 October 2026).

Prices are per million tokens, input and output separately. Price one ticket: count T-1004's prompt tokens with tiktoken (Module 2 covers counting), assume an output length, multiply.

%pip install -q -U langchain langgraph langchain-groq langchain-google-genai langchain-ollama

import os
from langchain.chat_models import init_chat_model

PROVIDER = "groq"   # "groq" or "gemini" in Colab; "ollama" on your own computer
MODELS = {"groq": "groq:openai/gpt-oss-20b", "gemini": "google_genai:gemini-3.5-flash-lite", "ollama": "ollama:llama3.2:3b"}
KEY = {"groq": "GROQ_API_KEY", "gemini": "GOOGLE_API_KEY", "ollama": None}[PROVIDER]
if KEY and KEY not in os.environ:
    from google.colab import userdata   # keys live in Colab's Secrets panel
    os.environ[KEY] = userdata.get(KEY)
llm = init_chat_model(MODELS[PROVIDER], temperature=0, max_retries=6)

def ask(model, messages):
    """Returns (reply, None), or (None, reason) after two refusals."""
    for attempt in range(2):                     # one retry: the next reply is usually fine
        try:
            return model.invoke(messages), None
        except Exception as err:
            last = err
    return None, f"provider_error: {type(last).__name__}"

Code walkthrough

• The standard setup cell (course-guide/02): PROVIDER picks the provider, llm is the client, max_retries=6 waits out "too many requests".
• ask is the course's guard: one retry, then a clear provider_error reason instead of a crash.

The slice of shared data this lesson needs: the categories and the 24 dev tickets, copied from course-guide/06.

CATEGORIES = ("billing", "cancellation", "account_access", "bug", "how_to", "feature_request")
CATEGORY_DEFINITIONS = {
    "billing": "Charges, invoices, prices, plan changes, and refund requests that do not end the plan.",
    "cancellation": "The customer wants to end a paid plan, with or without a refund.",
    "account_access": "Signing in, passwords, two-factor, SSO, and deleting a personal account.",
    "bug": "Something in Quillpath does not work as documented, including slowness and outages.",
    "how_to": "How to do something Quillpath already supports, or where to find a document.",
    "feature_request": "Something Quillpath does not do today that the customer would like.",
}
# (id, language, plan, subject, body, category, priority, gold article or None)
DEV = [
    ("T-1001", "en", "team", "Charged twice this month", "Hi, my card was charged 288 USD twice on 1 June for the Team plan (invoice INV-2026-004512). Please refund the duplicate. Ana Duarte, owner, Ardent Design", "billing", "high", "billing-refunds"),
    ("T-1002", "en", "business", "Switch to annual billing?", "We are on Business monthly with 8 seats. If we move to annual, what do we pay and when does it start?", "billing", "normal", "billing-plans"),
    ("T-1003", "en", "team", "VAT number on invoices", "Our accountant needs our VAT number on every invoice. Where do I add it, and can you reissue May's invoice?", "billing", "normal", "billing-invoices"),
    ("T-1004", "en", "team", "Refund for annual plan", "I bought the annual Team plan 20 days ago and we have decided not to use Quillpath. Can I get my money back? Yuki Sato, Studio Nami", "cancellation", "normal", "billing-refunds"),
    ("T-1005", "en", "business", "Cancel our subscription", "Please cancel our Business plan at the end of this month. We are moving to another tool.", "cancellation", "normal", "billing-refunds"),
    ("T-1006", "en", "free", "Forgot my password", "The reset email never arrived and I checked spam. My login is kofi.osei@example.com, phone +44 7700 900321.", "account_access", "high", "account-login"),
    ("T-1007", "en", "business", "SSO login loop", "Since this morning everyone in our team is sent back to the login page after signing in through our identity provider. Nobody can get in.", "account_access", "urgent", "account-sso"),
    ("T-1008", "en", "team", "Locked out", "I typed my password wrong a few times and now it says my account is locked. I have a client demo in an hour.", "account_access", "high", "account-login"),
    ("T-1009", "en", "team", "Add a teammate", "How do I invite a new person to our workspace, and will it change our bill?", "how_to", "low", "billing-plans"),
    ("T-1010", "en", "team", "Automations not firing", "Our rule 'move card to Done when checklist complete' stopped running yesterday. We use a lot of automations. Nothing changed on our side.", "bug", "normal", "boards-automations"),
    ("T-1011", "en", "business", "Export all our boards", "We need a full export of every board with comments for an audit. What formats do you support?", "how_to", "normal", "exports-data"),
    ("T-1012", "en", "business", "Slack posts duplicated", "Every card update now posts twice in our Slack channel.", "bug", "normal", "integrations-slack"),
    ("T-1013", "en", "team", "App crashes on start", "Since Friday's update the Android app closes as soon as I open it. Android 16. I already reinstalled and cleared the cache.", "bug", "normal", None),
    ("T-1014", "en", "free", "Dark mode please", "Would love a dark mode in the web app. My eyes would thank you.", "feature_request", "low", "feature-requests"),
    ("T-1015", "en", "business", "Is Quillpath down?", "Boards are loading very slowly for the whole team and most pages time out.", "bug", "urgent", "status-incidents"),
    ("T-1016", "en", "enterprise", "Data processing agreement", "Our legal team needs your DPA and the list of subprocessors before we renew.", "how_to", "normal", "data-privacy"),
    ("T-1017", "de", "team", "Doppelte Abbuchung", "Hallo, am 1. Juni wurden uns 60 USD zweimal abgebucht (Rechnung INV-2026-004530). Bitte erstatten Sie die doppelte Zahlung. Jonas Feldmann", "billing", "high", "billing-refunds"),
    ("T-1018", "es", "business", "No puedo entrar", "Desde ayer no puedo iniciar sesión. Dice que mi contraseña es incorrecta, pero la acabo de cambiar.", "account_access", "high", "account-login"),
    ("T-1019", "en", "team", "Delete my account", "Please delete my personal account and my data. I am leaving the company.", "account_access", "normal", "data-privacy"),
    ("T-1020", "en", "team", "Gantt view?", "Is there a timeline or Gantt view? I could not find one.", "feature_request", "low", "feature-requests"),
    ("T-1021", "en", "business", "Recurring cards", "Can I make a card that comes back every Monday?", "how_to", "low", "boards-automations"),
    ("T-1022", "en", "team", "Charged after cancelling", "I cancelled on 2 June but was charged again on 5 June. Why?", "billing", "high", None),
    ("T-1023", "en", "enterprise", "Contract question", "Our procurement team wants to change the liability clause in our Enterprise agreement. Who should they talk to?", "how_to", "normal", None),
    ("T-1024", "ja", "team", "請求書のダウンロード", "請求書をPDFでダウンロードするにはどうすればいいですか？", "how_to", "low", "billing-invoices"),
]
print(len(DEV), "dev tickets; T-1004 gold:", DEV[3][5])

Code walkthrough

• The six categories with their written definitions, and the 24 dev tickets exactly as Lena and Ravi labelled them. DEV[3] is T-1004.

%pip install -q tiktoken==0.14.0

import tiktoken

enc = tiktoken.get_encoding("o200k_harmony")
SYSTEM = ("You triage support tickets for Quillpath, a project-management app.\nCategories:\n"
          + "\n".join(f"- {c}: {d}" for c, d in CATEGORY_DEFINITIONS.items())
          + "\nReply with the category only.")
T1004 = DEV[3]
USER = f"Subject: {T1004[3]}\n\n{T1004[4]}"
n_in = len(enc.encode(SYSTEM + "\n" + USER))
N_OUT = 200   # assumed: a one-word label plus some hidden reasoning (section 3 measures it)

TIERS = {   # USD per 1,000,000 tokens (input, output), providers' pricing pages, checked 12 October 2026
    "frontier  claude-fable-5-1": (10.00, 50.00),
    "frontier  gpt-6-astra": (10.00, 50.00),
    "mid       claude-sonnet-5-5": (2.00, 10.00),
    "mid       gemini-3.8-flash": (0.75, 3.75),
    "small     gemini-3.5-flash-lite": (0.30, 2.50),
    "small     groq gpt-oss-20b": (0.075, 0.30),
}
def cost_per_ticket(price, n_in=n_in, n_out=N_OUT):
    return (n_in * price[0] + n_out * price[1]) / 1_000_000

print(n_in, "input tokens for T-1004")
for name, price in TIERS.items():
    c = cost_per_ticket(price)
    print(f"{name:32} {c * 100:6.3f} cents a ticket  {c * 1800:6.2f} USD a week  {'ok' if c <= 0.01 else 'OVER the cap'}")

Code walkthrough

• enc.encode splits text into tokens. SYSTEM is Lesson 4's triage prompt, asking for the category only.
• N_OUT is an assumption, labelled as one. TIERS holds list prices; closed models appear only as prices.
• cost_per_ticket is tokens times price per token; * 1800 is a week of tickets.

T-1004's prompt is 187 tokens. At 200 output tokens, a frontier model costs 1.187 cents a ticket, 1.2 times Joel's cap; the mid tier sits between 0.089 and 0.237 cents; the small tier costs 0.007 to 0.056 cents. The gap between the ends is about 160 times.

Tiers are a starting guess, not a verdict. The tier tells you the price; only your own tickets tell you whether the cheap model is good enough.

2. Proprietary APIs versus open-weight families (Essential)

Where the weights live decides who sees Quillpath's tickets, who can switch the model off and what you pay for. It is a business decision before it is a technical one.

A proprietary model is reached only through its maker's API: Gemini, Claude, GPT-6. You never see the weights; the maker can update or retire the model on its schedule. An open-weight model publishes its parameters under a licence, so anyone can host it: SmolLM2, Qwen2.5, Llama, and OpenAI's gpt-oss, which the course runs on Groq.

The course already uses both. Gemini is proprietary; Groq serves gpt-oss-20b, an open-weight model you could also run yourself. Tomasz's option is the third column: a small open model on Quillpath's own machine. Load three candidates.

%pip install -q transformers==5.19.0

import torch
from huggingface_hub import model_info
from transformers import AutoTokenizer, AutoModelForCausalLM

OPEN = ["HuggingFaceTB/SmolLM2-135M-Instruct", "HuggingFaceTB/SmolLM2-360M-Instruct", "Qwen/Qwen2.5-0.5B-Instruct"]
REV = dict(zip(OPEN, ["12fd25f77366fa6b3b4b768ec3050bf629380bac",       # pinned commits (course-guide/02)
                      "a10cc1512eabd3dde888204e902eca88bddb4951",
                      "7ae557604adf67be50417f59c2c2f167def9a775"]))
models = {}
for repo in OPEN:
    tok = AutoTokenizer.from_pretrained(repo, revision=REV[repo])
    model = AutoModelForCausalLM.from_pretrained(repo, revision=REV[repo]).eval()
    models[repo] = (tok, model)
    n = sum(p.numel() for p in model.parameters())
    licence = (model_info(repo).card_data or {}).get("license")
    print(f"{repo.split('/')[1]:22} {n / 1e6:4.0f}M parameters  {n * 2 / 1e9:4.2f} GB at 2 bytes  licence {licence}")

Code walkthrough

• OPEN lists three small open-weight models from two families, SmolLM2 and Qwen2.5. REV pins each to one commit of its files, so a later upload cannot change your results. from_pretrained downloads each at that commit; models keeps tokeniser and model together.
• n * 2 is memory for the weights at 2 bytes per parameter (Lesson 2). model_info reads the licence declared on the Hub page.

The three have 135M, 362M and 494M parameters, under a gigabyte each at 2 bytes, all Apache 2.0. Open weights move the bill: no fee per token, but you pay for the machine, the updates and the person on call when it breaks.

3. Reasoning models versus standard models (Essential)

Some models think before they answer, and you pay for the thinking. For a one-word label on T-1004, that can be most of the bill.

A reasoning model writes hidden working, reasoning tokens, before its visible answer. A standard model writes the answer straight away. gpt-oss-20b on Groq is a reasoning model; Groq lets you set reasoning_effort to low, medium or high. Gemini 3.5 Flash-Lite thinks at "minimal" by default, and Google bills "the sum of output tokens and thinking tokens" (both checked 12 October 2026).

Send T-1004 once and read the bill: how many output tokens were reasoning, and how many were the answer.

messages = [("system", SYSTEM), ("human", USER)]
reply, why = ask(llm, messages)
if reply is None:
    print("no answer:", why)                        # the next step handles a missing answer
else:
    usage = reply.usage_metadata or {}
    out = usage.get("output_tokens", 0)
    hidden = (usage.get("output_token_details") or {}).get("reasoning", 0)
    print("answer:", repr(reply.text.strip()[:40]), "| gold:", T1004[5])
    print(f"output tokens {out}: reasoning {hidden}, visible {out - hidden}")
    print(f"share of output spent on reasoning: {hidden / max(out, 1):.0%}")
    print(f"cost at {out} output tokens: {cost_per_ticket(TIERS['small     groq gpt-oss-20b'], n_out=out) * 100:.4f} cents (gpt-oss-20b prices)")

Code walkthrough

• ask sends T-1004; after a second failure the if prints the reason instead of crashing.
• usage_metadata counts the call's tokens; output_token_details["reasoning"] is the hidden part, when reported.
• The last line reprices the ticket with the measured output length.

When you run it, compare the reasoning count with the visible one, and the measured cost with the 200-token assumption. On Groq expect reasoning to outnumber the answer; on Gemini at its minimal default, few or none.

Now the same ticket on a standard model, SmolLM2-135M-Instruct, running locally.

tok, small = models[OPEN[0]]
chat = tok.apply_chat_template([{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER}],
                               tokenize=False, add_generation_prompt=True)
ids = tok(chat, return_tensors="pt", add_special_tokens=False)
with torch.no_grad():
    out_ids = small.generate(**ids, max_new_tokens=20, do_sample=False)
new = out_ids[0, ids["input_ids"].shape[1]:]
print(len(new), "output tokens, all visible:", repr(tok.decode(new, skip_special_tokens=True)))

Code walkthrough

• apply_chat_template wraps the same two messages in SmolLM2's chat format (Lesson 3).
• generate with do_sample=False writes the most likely tokens until the end-of-turn token or 20 tokens.
• Every token it writes is visible: there is no hidden working to pay for.

SmolLM2 wrote 20 tokens, all visible, and none of them a label: it began a letter, "Dear Yuki, I hope this message finds", and stopped at the 20-token limit. Free, fast and no use for triage. A reasoning model earns its hidden tokens on problems with steps; for a one-word label, test whether low effort or a standard model does as well before paying for thought.

4. Model cards and benchmark results (Recommended)

A model card is the maker's own description of a model. Read it before you download anything, and read its benchmark table as a claim, not a measurement of your task.

A model card states what the model is, how it was trained, its licence, its limits and its scores. A benchmark is a fixed public test with a scoring rule: IFEval checks instruction following, GSM8K grade-school maths, MMLU general knowledge. The SmolLM2-360M-Instruct card compares it with Qwen2.5-0.5B-Instruct and its own predecessor on eight of them.

import re
from huggingface_hub import ModelCard

def card_table(repo, heading="instruction model"):
    """The rows of the benchmark table under `heading` in a model card, as {metric: {model: score}}."""
    text = ModelCard.load(repo).text
    part = re.split(r"#+ " + heading, text, flags=re.I)[1].split("\n#")[0]
    lines = [l for l in part.splitlines() if l.startswith("|")]
    names = [c.strip() for c in lines[0].strip("|").split("|")][1:]
    rows = {}
    for l in lines[2:]:
        cells = [c.strip(" *") for c in l.strip("|").split("|")]
        rows[cells[0]] = {n: float(v) for n, v in zip(names, cells[1:])}
    return rows

card = card_table("HuggingFaceTB/SmolLM2-360M-Instruct")
for metric, scores in card.items():
    best = max(scores, key=scores.get)
    print(f"{metric:30} " + "  ".join(f"{v:6.2f}" for v in scores.values()) + f"   best: {best}")

Code walkthrough

• ModelCard.load downloads the card's text; re.split finds the instruction-model section and keeps its table lines.
• Each row becomes {model: score}; strip(" *") removes the bold marks. The loop prints each benchmark's winner.

On the card, SmolLM2-360M wins 5 of 8 rows, including IFEval at 41.0 against Qwen's 31.6. Qwen wins MT-Bench, BBH and GSM8K. Missing: support tickets, German or Japanese, and the prompt format, which Lesson 4 showed can move a small model by seven tickets.

Cards also disagree with each other. The SmolLM2-135M-Instruct card reports MT-Bench as 19.8, the 360M card as 3.66: the same benchmark name, numbers on visibly different scales, and neither card explains. Before comparing two cards, check the scale, the settings and who ran the test.

5. Why leaderboard rank rarely transfers (Recommended)

A rank is an average over someone else's tasks. Your task is one point, and one point can sit anywhere.

A leaderboard ranks models on one benchmark or a blend. Rank fails to transfer for three reasons. The test is not your task. The entry may not be the model you get, as with Maverick. And makers tune for the board: "The Leaderboard Illusion" estimates that limited extra arena data gave "relative performance gains of up to 112% on the arena distribution".

Break it: pick the model the card ranks first

By IFEval, the instruction-following benchmark, the order is SmolLM2-360M, then Qwen2.5-0.5B, then SmolLM2-135M. Score all three on the 24 dev tickets with Lesson 4's label scoring: the probability each model gives each category name.

def label_probs(repo, row):
    """Probability of each category name as the reply, renormalised over the six (Lesson 4)."""
    tok, model = models[repo]
    chat = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": f"Subject: {row[3]}\n\n{row[4]}"}]
    p = tok(tok.apply_chat_template(chat, tokenize=False, add_generation_prompt=True), add_special_tokens=False).input_ids
    label_ids = [tok(c, add_special_tokens=False).input_ids for c in CATEGORIES]
    longest = max(map(len, label_ids))
    batch = torch.tensor([p + l + [0] * (longest - len(l)) for l in label_ids])   # pad after each label
    with torch.no_grad():
        logp = torch.log_softmax(model(batch).logits, dim=-1)
    scores = torch.stack([sum(logp[i, len(p) - 1 + j, t] for j, t in enumerate(l)) for i, l in enumerate(label_ids)])
    return dict(zip(CATEGORIES, torch.softmax(scores, 0).tolist()))

IFEVAL = {r: card_table(r)["IFEval (Average prompt/inst)"][r.split("/")[1]] for r in OPEN[:2]}
IFEVAL[OPEN[2]] = card["IFEval (Average prompt/inst)"]["Qwen2.5-0.5B-Instruct"]
gold = {row[0]: row[5] for row in DEV}
picks = {}
for repo in sorted(OPEN, key=IFEVAL.get, reverse=True):          # 1 to 5 minutes per model on a CPU
    probs = {row[0]: label_probs(repo, row) for row in DEV}
    picks[repo] = {t: max(p, key=p.get) for t, p in probs.items()}
    k = sum(picks[repo][t] == gold[t] for t in gold)
    top = max(CATEGORIES, key=list(picks[repo].values()).count)
    print(f"{repo.split('/')[1]:22} IFEval {IFEVAL[repo]:4.1f}  dev {k:2} of 24  T-1004: {picks[repo]['T-1004']:12}"
          f"  most common: {top} x{list(picks[repo].values()).count(top)}")

Code walkthrough

• label_probs is Lesson 4's label scoring, for any of the three models.
• IFEVAL reads each IFEval score from the cards (Qwen's from the 360M card).
• The loop runs the models in card order and reports right answers, T-1004 and the most common label.

The card's winner came last. SmolLM2-360M scored 5 of 24 and called 21 tickets billing, below the always-how_to baseline of 6. SmolLM2-135M scored 13 and Qwen 12. T-1004 got three different labels: how_to, billing and cancellation.

Why? From outside we cannot tell what in this prompt suits one model's training and not another's. That is the point: the IFEval rank is real, on IFEval, and says little about Quillpath's tickets.

6. Model selection framework (Essential)

Without a written rule, the selection is whoever argued last. With one, you can rerun it when a price changes or a model is retired, which in this field is every few months.

A selection framework is a fixed order of questions. First, hard constraints that rule a model out: spend cap, where the data may go, languages. Second, a quality bar measured on your own labelled set, with an interval. Third, among the models that pass, the cheapest, with a fallback from a different provider.

Write Quillpath's version as code. The bar is Lesson 4's: the low end of a model's interval must clear the high end of the always-how_to baseline's, 45%. A model nobody has measured cannot pass; it goes on a "measure first" list.

from math import sqrt

def wilson(k, n, z=1.96):
    """95% interval for a success rate of k out of n (Lesson 4)."""
    p = k / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centre - half, centre + half

BAR = wilson(6, 24)[1]   # high end of the always-how_to baseline
right = {r: sum(picks[r][t] == gold[t] for t in gold) for r in OPEN}
CANDIDATES = {   # name: (where it runs, (input, output) USD per 1M tokens, right on 24 dev tickets or None)
    "gemini-3.5-flash-lite": ("Google API", (0.30, 2.50), 20),        # Lesson 4, Gemini run, 10 October 2026
    "groq gpt-oss-20b": ("Groq API", (0.075, 0.30), None),           # not measured yet
    "gemini-3.8-flash": ("Google API", (0.75, 3.75), None),
    **{r.split("/")[1]: ("own server", (0.0, 0.0), right[r]) for r in OPEN},
}
CAP = 0.01   # Joel: USD of model fees per ticket

def select(candidates, must=lambda where: True):
    passed, measure_first = [], []
    for name, (where, price, k) in candidates.items():
        if not must(where) or cost_per_ticket(price) > CAP:
            continue                                            # 1. hard constraints
        if k is None:
            measure_first.append(name)                          # no evidence, no selection
        elif wilson(k, 24)[0] > BAR:                            # 2. quality bar on our tickets
            passed.append((cost_per_ticket(price), name))
    return sorted(passed), measure_first                        # 3. cheapest first

passed, measure_first = select(CANDIDATES)
print(f"bar: low end above {BAR:.0%}")
print("passed, cheapest first:", [(n, f"{c * 100:.3f} cents") for c, n in passed])
print("measure first:", measure_first)

Code walkthrough

• BAR is computed, not typed. right holds section 5's scores.
• CANDIDATES records where each model runs, its price and its score; None means unmeasured.
• select applies the three steps in order; must is a constraint on where the model runs.

Only gemini-3.5-flash-lite passes, at 0.056 cents a ticket by section 1's assumption. None of the three local models clears the bar. gpt-oss-20b is about 7.5 times cheaper but unmeasured, so it goes first on the "measure first" list, with gemini-3.8-flash.

The written selection for Maya: Gemini 3.5 Flash-Lite for now; measure gpt-oss-20b next (Module 4's harness); a second provider as fallback (Module 13); rerun when prices change. Prices dated 12 October 2026 will go stale first, then names: Google already limits Gemini 2.5 to earlier users.

In production

Teams keep the provider and model name in one setting, as MODELS does, so a switch is one line behind a test. They pin model versions, watch deprecation pages and rerun their own test set before any switch, as you will for Tomasz in week 13. Leaderboards decide which models to try, never which to ship. Closed models are added the same way:

MODELS_CLOSED = {   # not run here: needs langchain-openai or langchain-anthropic and a paid key
    "openai": "openai:gpt-6-luna",          # OpenAI's "most efficient model", checked 12 October 2026
    "anthropic": "anthropic:claude-haiku-5-5",
}
# llm = init_chat_model(MODELS_CLOSED["anthropic"], temperature=0, max_retries=6)
print(sorted(MODELS) + sorted(MODELS_CLOSED))

Code walkthrough

• Two more provider:model strings in LangChain's format. Nothing else in the lesson would change. The call line stays commented out, because the course runs only open and free-tier models.

What you added to the Quillpath support desk

• A price per ticket on every tier, against Joel's one-cent cap.
• A measured reasoning-token count for T-1004 on the course model.
• Three small open models scored on the dev set, and evidence that a card's rank did not predict ours.
• select, the written selection rule, and its first answer: Gemini 3.5 Flash-Lite, with gpt-oss-20b to measure next.

Check yourself

1. Why is "the top model on LM Arena" not an answer to "which model should triage Quillpath's tickets"?
2. gpt-oss-20b is served by Groq through an API. Is it proprietary?
3. A reasoning model returns a one-word label. What are you paying for besides that word?
4. The SmolLM2-360M card wins five of eight benchmarks against Qwen2.5-0.5B. Why did it score lowest here?
5. Why does select refuse to pick a model with no measured score, even a cheap one?

1. The arena measures votes on other people's chats, the entry may not be the released model, and makers tune for it.
2. No. Its weights are open (Apache 2.0); Groq is one host. You could run the same weights yourself.
3. The reasoning tokens, billed as output, even though you never see them.
4. The benchmarks are not Quillpath's task or prompt; on our prompt it called 21 tickets billing.
5. Without a score there is no evidence it clears the bar; price alone would pick the cheapest model whatever its quality.

Exercise

Anika asks: "What if customer messages must never leave Quillpath's servers?" Before running it, predict what select returns with that constraint. Then run it and write one sentence for Maya on what the constraint costs.

local_passed, local_measure = select(CANDIDATES, must=lambda where: where == "own server")
assert local_passed == [] and local_measure == [], "a local model passed: check the bar and the scores"
best_local = max(right, key=right.get)
lo, hi = wilson(right[best_local], 24)
print(f"no local model passes; best is {best_local.split('/')[1]}: {right[best_local]} of 24, interval {lo:.0%} to {hi:.0%}")
print(f"it needs its low end above {BAR:.0%}: try a larger open model, a better prompt (Module 4) or fine-tuning (Module 9)")

Code walkthrough

• must now admits only models that run on Quillpath's own machine. The assert checks your prediction: none of the three passes, and none is unmeasured.
• The last lines show how far the best local model is from the bar, and the three ways the course will try to close the gap.

Key terms

Sources

• Kyle Wiggers, TechCrunch, "Meta's vanilla Maverick AI model ranks below rivals on a popular chat benchmark", 11 April 2025. https://techcrunch.com/2025/04/11/metas-vanilla-maverick-ai-model-ranks-below-rivals-on-a-popular-chat-benchmark/
• Singh, Nan, Wang, D'Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee, Hooker, "The Leaderboard Illusion", 29 April 2025. https://arxiv.org/abs/2504.20879
• Anthropic, "Models overview", checked 12 October 2026. https://platform.claude.com/docs/en/about-claude/models/overview
• OpenAI, "Models", checked 12 October 2026. https://developers.openai.com/api/docs/models
• Google, "Gemini models" and "Gemini Developer API pricing", checked 12 October 2026. https://ai.google.dev/gemini-api/docs/models and https://ai.google.dev/gemini-api/docs/pricing
• Google, "Gemini thinking", checked 12 October 2026. https://ai.google.dev/gemini-api/docs/thinking
• Groq, "Supported models" and "Reasoning", checked 12 October 2026. https://console.groq.com/docs/models and https://console.groq.com/docs/reasoning
• Hugging Face, SmolLM2-135M-Instruct and SmolLM2-360M-Instruct model cards, checked 12 October 2026. https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct
• Qwen team, Qwen2.5-0.5B-Instruct model card, checked 12 October 2026. https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct