Lesson 2: Evaluation

Why evaluation is a separate discipline

A test asserts that a specific input produces a specific output. That works when the function is deterministic and fails immediately for a model call, where the same input produces different text each time, none of it wrong.

Evaluation measures quality across many examples rather than asserting correctness on one. It answers a different question: not "is this right" but "is this better than what we had", and the answer is a distribution rather than a boolean.

This is the bottleneck in most AI systems. Teams ship on the basis of trying a few prompts and liking the output, then cannot tell whether the next change helped, so quality drifts in a direction nobody chose.

Building an eval dataset

Module 9 introduced the labelled query set for retrieval. This extends it to the whole system.

@dataclass(frozen=True)
class EvalCase:
    id: str
    input: str
    category: str                          # for slicing results

    # Expectations, whichever apply to the task
    gold_answer: str | None = None
    gold_chunk_ids: list[str] = field(default_factory=list)
    must_contain: list[str] = field(default_factory=list)
    must_not_contain: list[str] = field(default_factory=list)
    should_abstain: bool = False
    rubric: str | None = None

Where cases come from, in order of value.

Real inputs from production logs. Synthetic cases are systematically easier: better spelled, more grammatical, and more likely to have an answer than what users actually send.

Every failure you have already found. A bug fixed without a case added is a bug that will return.

Deliberately hard categories: multi-hop questions, exact-identifier lookups, ambiguous questions, and questions with no answer in the corpus.

That last category is the one most often missing, and it measures the behaviour that matters most. A system confidently answering what it cannot know is worse than one that abstains, and should_abstain is how you check it.

Size. Fifty to a hundred cases is enough to detect meaningful differences and cheap enough to run often. A thousand cases you run quarterly is worth less than eighty you run on every prompt change.

Slice by category from the start. An average of 0.82 can hide 0.95 on simple lookups and 0.41 on multi-hop, and only the slice tells you what to fix.

Task-appropriate metrics

Choose the metric the task actually has, rather than defaulting to one.

Exact match works for closed outputs: a classification label, an extracted enum, a boolean. Trivially computable and unambiguous.

F1 balances precision and recall for extraction with multiple items, such as pulling ingredients from a recipe.

def f1(predicted: set[str], gold: set[str]) -> float:
    if not predicted and not gold:
        return 1.0
    if not predicted or not gold:
        return 0.0
    tp = len(predicted & gold)
    precision = tp / len(predicted)
    recall = tp / len(gold)
    return 2 * precision * recall / (precision + recall) if tp else 0.0

Recall@k for retrieval, from Module 9, which sets the ceiling on everything downstream.

Rubric scores for open-ended generation where no single correct answer exists. This is where a judge comes in, covered below.

A cost and latency figure alongside every quality number. Module 10 established the ceilings, and quality measured without them lets a change that improves quality by two percent and triples cost look like a win.

Assertion-based checks

The cheapest and most underused tier. Many quality requirements are deterministic properties of the output.

def check_assertions(case: EvalCase, answer: GroundedAnswer) -> list[str]:
    """Return failure descriptions. Empty list means all checks passed."""
    failures: list[str] = []

    for required in case.must_contain:
        if required.lower() not in answer.text.lower():
            failures.append(f"missing required content: {required!r}")

    for forbidden in case.must_not_contain:
        if forbidden.lower() in answer.text.lower():
            failures.append(f"contains forbidden content: {forbidden!r}")

    if case.should_abstain and not is_abstention(answer):
        failures.append("should have abstained but produced an answer")

    if answer.citations and not all(c in case.available_sources for c in answer.citations):
        failures.append("cited a source not present in the context")

    if len(answer.text.split()) > MAX_ANSWER_WORDS:
        failures.append(f"answer exceeds {MAX_ANSWER_WORDS} words")

    return failures

These run in microseconds, cost nothing, and never disagree with themselves. Write every assertion you can before reaching for a judge, because a judge is expensive, variable, and needed only for the part that genuinely requires reading.

The citation check is the semantic validation from Module 6 promoted into the eval tier, and it catches a real failure: an answer that looks well-sourced and cites something that was never provided.

LLM-as-judge

For qualities that cannot be checked programmatically, such as whether an answer is faithful to its sources or helpful for the question, use a model to score it.

JUDGE_SYSTEM = """You evaluate answers against source material.

Score faithfulness from 1 to 5:
5: every claim is directly supported by the sources
4: all claims supported, minor unsupported elaboration
3: main claims supported, some unsupported detail
2: significant claims not supported by the sources
1: contradicts the sources or is largely invented

Judge only faithfulness to the sources. Do not judge style, length, or
whether the answer is what you would have written.

Return JSON: {"score": <1-5>, "reasoning": "<one sentence>", "unsupported_claims": [...]}"""

Rubric design determines whether the judge is useful.

Define each score point concretely, so that a rubric saying "5 is excellent" produces scores that mean nothing.

Judge one dimension per call. A judge asked to rate faithfulness, helpfulness, and tone at once produces a blended number that hides which one failed.

State explicitly what not to judge. Judges drift toward rating what they would have written.

Require reasoning before the score, and ask for the specific unsupported claims, which makes the judgement auditable and improves accuracy.

Pointwise versus pairwise. Pointwise scores one output on a scale, which is what you need for absolute thresholds and tracking over time. Pairwise asks which of two outputs is better, which is more reliable because relative judgements are easier than absolute ones, and is the right choice when comparing two prompt versions.

If using pairwise, swap the order and run it twice, because of position bias.

Known judge biases, all of which are measurable.

Position bias: preferring the first or second option regardless of content. Counter by swapping and averaging.

Verbosity bias: preferring longer answers. Counter by stating explicitly that length is not a criterion, and by checking whether your judge's scores correlate with answer length.

Self-preference: preferring output from the same model family. Counter by using a different model as judge where practical.

Sycophancy: agreeing with any framing in the prompt. Counter by not telling the judge which output came from the new version.

Judge calibration

A judge you have not calibrated is a number you cannot trust.

The procedure. Label thirty to fifty outputs yourself against the same rubric. Run the judge on the same outputs. Compare.

@dataclass(frozen=True)
class CalibrationReport:
    exact_agreement: float          # identical scores
    within_one: float               # within one point on the scale
    judge_mean: float
    human_mean: float
    disagreements: list[Disagreement]


def calibrate(human: dict[str, int], judge: dict[str, int]) -> CalibrationReport:
    ids = sorted(set(human) & set(judge))
    exact = sum(1 for i in ids if human[i] == judge[i]) / len(ids)
    close = sum(1 for i in ids if abs(human[i] - judge[i]) <= 1) / len(ids)
    return CalibrationReport(
        exact_agreement=exact,
        within_one=close,
        judge_mean=mean(judge[i] for i in ids),
        human_mean=mean(human[i] for i in ids),
        disagreements=[
            Disagreement(i, human[i], judge[i]) for i in ids if abs(human[i] - judge[i]) > 1
        ],
    )

Reading it. Within-one agreement above roughly 0.85 means the judge is usable for tracking changes. A systematic offset, where the judge is consistently a point higher, is tolerable for comparing versions and not for absolute thresholds. Scattered disagreement means the rubric is ambiguous.

Read the disagreements individually. They almost always reveal a rubric problem rather than a judge problem, and fixing the rubric is what improves agreement. Calibrate, fix, recalibrate.

Recalibrate when anything changes: the judge model, the rubric, or the task. A judge calibrated six months ago on a different model is not evidence.

[IMAGE PROMPT M11-2
Purpose: Show how judge scores are compared against human labels and what different disagreement patterns mean.
Visual type: Scatter comparison with three annotated patterns.
Prompt: A clean educational diagram showing three small square scatter plots side by side, each with a horizontal axis labelled "human score 1 to 5" and a vertical axis labelled "judge score 1 to 5", and a light diagonal line labelled "perfect agreement". The left plot is headed "Calibrated" and shows points clustered tightly along the diagonal, annotated beneath "within one point: 0.91, usable". The middle plot is headed "Systematic offset" and shows points clustered along a line parallel to and above the diagonal, annotated beneath "judge scores consistently high, fine for comparison, not for thresholds". The right plot is headed "Ambiguous rubric" and shows points scattered widely with no clear relationship, annotated beneath "fix the rubric, not the judge". A note beneath all three reads "read the individual disagreements, they usually reveal a rubric problem".
Required elements: Three labelled plots with identical axes and diagonal reference lines, three distinct point patterns, per-plot annotations, the shared note.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, small uniform point markers.
Layout: Three equal square plots side by side, annotations beneath each, shared note at the bottom.
Text labels: "human score 1 to 5", "judge score 1 to 5", "perfect agreement", "Calibrated", "Systematic offset", "Ambiguous rubric", "within one point: 0.91, usable", "judge scores consistently high, fine for comparison, not for thresholds", "fix the rubric, not the judge", "read the individual disagreements, they usually reveal a rubric problem".
Aspect ratio: 16:9
Accessibility: Distinguish the three patterns by point arrangement and text annotation rather than colour alone.
Avoid: Vendor logos, decorative elements, tiny text, watermarks.
Alt text: Three scatter plots comparing judge scores against human labels, showing tight agreement along the diagonal, a systematic upward offset, and wide scatter indicating an ambiguous rubric.
END IMAGE PROMPT]

Variance, sample size, and the noise floor

Run the same configuration twice before believing any improvement.

async def measure_noise_floor(cases: list[EvalCase], pipeline: Pipeline, *, runs: int = 3) -> float:
    """Run the identical configuration several times and report the spread."""
    scores = [await run_eval(cases, pipeline) for _ in range(runs)]
    return max(scores) - min(scores)

That spread is your noise floor. Any change producing a smaller improvement has produced no measurable improvement, whatever the number says.

The arithmetic is unforgiving. On forty cases, one case flipping is 2.5 percentage points. With generation temperature above zero, several will flip between identical runs. A reported gain of three points on forty cases is within noise and means nothing.

Three ways to make results trustworthy. More cases, since noise falls roughly with the square root of the count. Temperature at zero for evaluation, which reduces but does not eliminate variation. And averaging several runs per configuration rather than one.

Report the spread alongside the number. "0.84 plus or minus 0.03 over three runs" is a result. "0.84" is a number that might be 0.81.

[IMAGE PROMPT M11-3
Purpose: Show why a reported improvement smaller than run-to-run variance is not a result.
Visual type: Dot plot of repeated runs with an overlaid comparison.
Prompt: A clean educational chart with a vertical axis labelled "eval score" and two labelled groups along the horizontal axis: "Configuration A (baseline)" and "Configuration B (new prompt)". Above group A, five small dots are plotted at slightly different heights, with a bracket spanning them labelled "same config, five runs, spread 0.05". Above group B, five dots are plotted at heights overlapping substantially with group A's range, with its own bracket labelled "spread 0.04". A dashed horizontal line marks each group's mean, labelled "A mean 0.82" and "B mean 0.84". A callout to the right reads "reported gain 0.02, smaller than the spread, not a result". Beneath the chart, a second small panel shows the same comparison with tighter spreads and a wider gap between means, annotated "gain 0.09 with spread 0.02, this is a result".
Required elements: Two groups of repeated-run dots with visible overlap, per-group spread brackets, mean lines with labels, the not-a-result callout, a contrasting second panel showing a genuine result.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, uniform dot markers.
Layout: Main chart occupying the upper two thirds, contrasting panel beneath.
Text labels: "eval score", "Configuration A (baseline)", "Configuration B (new prompt)", "same config, five runs, spread 0.05", "spread 0.04", "A mean 0.82", "B mean 0.84", "reported gain 0.02, smaller than the spread, not a result", "gain 0.09 with spread 0.02, this is a result".
Aspect ratio: 16:9
Accessibility: Show spread with explicit brackets and numeric labels rather than relying on visual density or colour.
Avoid: Vendor logos, decorative elements, tiny text, watermarks.
Alt text: Dot plot showing five repeated runs of two configurations whose score ranges overlap, with a reported two point gain smaller than the five point run-to-run spread, contrasted with a second panel where a nine point gain exceeds a two point spread.
END IMAGE PROMPT]

Error analysis

Read the failures before changing anything. This is the highest-value habit in the module and the one most often skipped in favour of trying an idea.

The procedure: take fifty failures, read each one, and write down what went wrong in your own words. Do not propose fixes while reading. Then cluster the descriptions and count the clusters.

@dataclass(frozen=True)
class FailureNote:
    case_id: str
    what_happened: str          # written by a human, in plain words
    cluster: str | None = None  # assigned after reading all of them

Clustering after reading rather than during is deliberate, because deciding categories up front makes you sort failures into the buckets you expected rather than discovering the ones you did not.

What usually emerges is that the failures are not what you assumed. A team convinced their prompt is too vague reads fifty failures and finds forty are retrieval misses, which is the Module 9 diagnosis applied systematically.

Then fix the largest cluster, and only that one. Fixing three things at once and measuring an improvement tells you the combination worked, not which part did.

Golden sets and regression suites

Once a case passes, it should keep passing.

async def test_regression_suite_holds(eval_pipeline) -> None:
    """Cases that previously passed must not start failing."""
    report = await run_eval(load_golden_cases(), eval_pipeline)
    regressions = [r for r in report.results if r.case_id in PREVIOUSLY_PASSING and not r.passed]
    assert not regressions, f"regressed on: {[r.case_id for r in regressions]}"

Prompt changes are where this earns its place. A prompt edit fixing one failure mode frequently breaks another, because prompt effects are not local. Without a regression suite you fix a bug and ship a different one.

Add a case for every bug you fix, in the same commit as the fix. That single habit converts your eval set from something you built once into something that grows with what you have learned.

Online signals and A/B testing

Offline evaluation measures what you thought to measure. Production tells you what users experience.

Signals worth capturing. Explicit feedback such as thumbs, which is sparse and biased toward extremes. Implicit signals such as whether the user copied the answer, edited it, immediately rephrased the question, or abandoned the session. Escalation to a human, which is the strongest negative signal you get. And your own abstention rate, since a rising one means retrieval or the corpus has degraded.

A/B testing compares two versions on real traffic.

def variant_for(user_id: str, experiment: str) -> str:
    """Stable assignment, so a user stays in one arm for the experiment."""
    digest = hashlib.sha256(f"{experiment}:{user_id}".encode()).hexdigest()
    return "treatment" if int(digest[:8], 16) % 100 < 50 else "control"

Assignment must be stable per user, or someone sees both versions and their behaviour measures neither.

Run offline evaluation first. A/B tests are slow, need traffic, and expose users to the worse version. Use them to confirm what offline evaluation suggested and to measure the effects offline cannot see, such as whether users actually engage differently.

Threshold gates

Evaluation earns its cost by blocking bad changes automatically.

@dataclass(frozen=True)
class QualityGate:
    metric: str
    minimum: float | None = None
    maximum: float | None = None
    slice: str | None = None


GATES = [
    QualityGate("faithfulness_mean", minimum=4.0),
    QualityGate("assertion_pass_rate", minimum=0.95),
    QualityGate("recall_at_5", minimum=0.80),
    QualityGate("abstention_correctness", minimum=0.90, slice="unanswerable"),
    QualityGate("cost_per_request_usd", maximum=0.012),
    QualityGate("p95_latency_ms", maximum=4000),
]

Three properties of a good gate set. Quality, cost, and latency together, because a change improving one at the expense of the others should not pass silently, which is the connection Module 10 ended on.

Slice-specific gates, since the unanswerable slice above is the abstention behaviour an overall average would hide.

Floors set below current performance, not at it. A gate at exactly today's number fails on noise and gets disabled. Set it where a real regression would trip it and normal variation would not.

Concept check. Your judge scores rose from 3.9 to 4.3 after a prompt change, on 40 cases, single run. Your colleague wants to ship. What do you say?

Answer

That you do not yet know whether it is real. On 40 cases a single case moving two points shifts the mean by 0.05, so a 0.4 gain could be eight cases genuinely improving or could be run-to-run variation, and one run cannot distinguish them.

Run both configurations three times each and compare the means against the spread. If the gain exceeds the noise floor, it is a result.

Two further checks before shipping. Look at the per-category slice, because a gain concentrated in easy cases while hard cases regressed is not an improvement. And check the cost and latency numbers alongside, since a prompt change that adds few-shot examples improves quality and raises cost on every request.

Worth adding: read the cases that got worse. A change improving the average while breaking a category you care about is a common and avoidable mistake.