Lesson 5: CI Layering

Fast suite per commit, eval nightly, cost-bearing gated

Not every check can run on every commit. The layering matches cost and duration against how quickly you need the feedback.

# Illustrative CI configuration. Adapt to your platform.
jobs:
  fast:
    # Every commit and pull request. Target: under two minutes.
    steps:
      - run: uv sync --locked
      - run: uv run ruff check .
      - run: uv run ruff format --check .
      - run: uv run mypy src
      - run: uv run pytest -m "not integration and not live and not eval"

  integration:
    # Every commit. Target: under ten minutes.
    services: [postgres, redis]
    steps:
      - run: uv run pytest -m integration

  eval:
    # Nightly, and on demand for prompt or model changes.
    steps:
      - run: uv run pytest -m eval --eval-report=eval.json
      - run: uv run python -m recipe_extractor.eval.gate eval.json

  live:
    # Weekly, and manually before a provider migration. Costs money.
    steps:
      - run: uv run pytest -m live

The fast suite must stay fast. Under two minutes is a target worth defending, because a suite taking fifteen minutes gets skipped, and a skipped suite is not a suite. When it slows, find what moved into it rather than accepting the new duration.

Note that the fast tier includes ruff and mypy from Module 1. Type checking catches a class of error before any test runs, and it is faster than the tests.

The eval tier runs nightly because it costs money and takes minutes, and because its results are statistical rather than binary. It should also run on demand for any prompt, model, or retrieval change, which is exactly when a quality regression is likely.

The live tier is gated and scheduled, protecting against the recorded-response staleness from Lesson 1: a provider changing their response shape breaks production while every recorded test still passes, and only a real call catches it.

Gates and reporting

def apply_gates(report: EvalReport, gates: list[QualityGate]) -> GateResult:
    """Fail the build on regressions, and always print the trend."""
    failures: list[str] = []
    for gate in gates:
        value = report.metric(gate.metric, slice=gate.slice)
        if gate.minimum is not None and value < gate.minimum:
            failures.append(f"{gate.metric} = {value:.3f}, floor is {gate.minimum}")
        if gate.maximum is not None and value > gate.maximum:
            failures.append(f"{gate.metric} = {value:.3f}, ceiling is {gate.maximum}")
    return GateResult(passed=not failures, failures=failures, report=report)

Report the trend, not just the verdict. A metric moving from 0.88 to 0.81 against a floor of 0.80 passes and is the most important thing in the run. Print the delta against the previous run for every gated metric, and the pass or fail becomes a floor under a conversation rather than the whole conversation.

Set floors below current performance. A gate at exactly today's value fails on noise, and a gate that fails on noise gets disabled within a fortnight. Use the noise floor from Lesson 2 to place it: below current performance by more than the measured spread.

Gate on cost and latency too. Module 10's cost ceilings belong in the same job as the quality floors, because that is the pairing that stops a quality improvement shipping at three times the price.

Concept check. Your nightly eval has failed four times in two weeks, each time on a different metric, each time passing when re-run. What is wrong and what do you change?

Answer

The gates are set too close to current performance, so ordinary run-to-run variation crosses them. The evidence is in the pattern: a different metric each time and passing on re-run is exactly what noise looks like, whereas a real regression fails consistently and on the same metric.

Measure the noise floor by running the identical configuration several times and recording the spread for each metric. Then set each floor below current performance by more than that spread, so a genuine regression trips it and normal variation does not.

Two supporting changes. Increase the case count if the spread is large, since noise falls roughly with the square root of the number of cases. And average several runs per nightly job rather than taking one, which reduces the variance of the number being gated.

The risk of doing nothing is worse than a noisy gate: a suite that cries wolf gets disabled, and then a real regression ships unnoticed.