Lesson 4: Cost Control

Per-user quotas, hard caps, and kill switches

Prompt caching and cache-aware layout

Prompt caching lets a provider reuse the computation for a prompt prefix it has seen recently. Where supported, it is the single largest cost lever available without changing behaviour.

The mechanism sets the requirement: the prefix must be byte identical. Caching works on the beginning of the prompt, and any difference invalidates everything from that point on.

That produces one rule: stable content first, volatile content last.

# Cache hostile: the timestamp changes every request, invalidating everything after it
messages = [
    Message(role="system", content=f"Current time: {datetime.now()}\n\n{SYSTEM_PROMPT}"),
    Message(role="user", content=f"{retrieved_context}\n\n{user_question}"),
]

# Cache friendly: everything stable is at the front
messages = [
    Message(role="system", content=SYSTEM_PROMPT),          # stable across all requests
    Message(role="user", content=TOOL_DOCUMENTATION),        # stable
    Message(role="assistant", content=FEW_SHOT_EXAMPLES),    # stable
    Message(role="user", content=retrieved_context),         # varies per query
    Message(role="user", content=f"[{datetime.now()}] {user_question}"),  # varies always
]

The ordering advice from Module 3 was about attention. This is the same ordering advice for a different reason, and the two agree.

What breaks caching in practice, and all of these are common. A timestamp or request identifier at the top. Tool definitions serialised in a different order between requests, since dictionary ordering must be stable. Retrieved context placed before the system prompt. A user identifier or personalisation injected early. And a system prompt assembled by string formatting where a value occasionally differs.

Minimum sizes apply. Providers typically require a prefix above some token threshold before caching engages, often around a thousand tokens. This produces a counterintuitive optimisation: a prompt slightly below the threshold can be made cheaper by making it longer, because crossing the threshold enables caching on every subsequent request. Verify the threshold for your provider before relying on this.

Caches expire, often within minutes, and are typically scoped to your account or workspace and to a specific model. A cache built for one model is not available to another, which matters for the routing in the next section.

Measure the hit rate. Providers report cached token counts in the usage object, and Module 8 noted that for streams these arrive in the final chunk.

def cache_hit_rate(records: list[CostRecord]) -> float:
    """Fraction of input tokens served from cache."""
    total_input = sum(r.usage.input_tokens for r in records)
    cached = sum(r.usage.cached_input_tokens for r in records)
    return cached / total_input if total_input else 0.0

A hit rate near zero on a system with a large stable prefix means something is invalidating it, and the layout above is where to look.

[IMAGE PROMPT M10-5
Purpose: Show why volatile content at the start of a prompt destroys caching, and how reordering restores it.
Visual type: Two-panel prompt structure comparison with cache boundary marked.
Prompt: A clean educational comparison with two stacked panels, each showing a vertical stack of prompt segments reading top to bottom. The upper panel is headed "Cache hostile" and shows segments in order: a thin segment labelled "timestamp, changes every request", then "system prompt", "tool definitions", "few-shot examples", "retrieved context", "user question". A vertical bracket on the left spans from the timestamp downward, labelled "all invalidated by the first change", with an annotation reading "0% cache hit". The lower panel is headed "Cache friendly" and shows segments in order: "system prompt", "tool definitions", "few-shot examples", then a horizontal dividing line labelled "cache boundary", then "retrieved context", "user question with timestamp". A vertical bracket spans the three segments above the boundary labelled "stable prefix, cached and reused", with an annotation reading "high cache hit". A note beneath reads "caching matches a byte-identical prefix, so one early change costs everything after it".
Required elements: Two segment stacks with the same content in different orders, a bracket showing the invalidated region above and the cached region below, an explicit cache boundary line in the lower panel, hit rate annotations, the note.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, segments as labelled horizontal bands.
Layout: Two panels stacked vertically, each reading top to bottom, brackets on the left, annotations on the right.
Text labels: "Cache hostile", "Cache friendly", "timestamp, changes every request", "system prompt", "tool definitions", "few-shot examples", "retrieved context", "user question", "user question with timestamp", "cache boundary", "all invalidated by the first change", "stable prefix, cached and reused", "0% cache hit", "high cache hit", "caching matches a byte-identical prefix, so one early change costs everything after it".
Aspect ratio: 4:3
Accessibility: Mark the cached and invalidated regions with brackets and text labels rather than colour alone.
Avoid: Vendor logos, real prompt text, decorative elements, tiny text, watermarks.
Alt text: Comparison of two prompt layouts, one with a changing timestamp at the top invalidating the entire cache, and one with stable system content first and volatile content after a cache boundary, achieving a high cache hit rate.
END IMAGE PROMPT]

Cost-aware model routing and escalation

Not every request needs your most capable model.

async def route_and_generate(task: Task) -> Completion:
    """Try the cheap model first, escalate only when it is not good enough."""
    if task.complexity == "simple":
        return await cheap_provider.generate(task.messages, model=CHEAP_MODEL)

    result = await cheap_provider.generate(task.messages, model=CHEAP_MODEL)
    if await is_acceptable(result, task):
        return result

    logger.info("escalating", extra={"task_id": task.id, "reason": "quality_gate"})
    return await strong_provider.generate(task.messages, model=STRONG_MODEL)

Escalation triggers, in order of reliability. A structured output that fails validation is an unambiguous signal. A tool call the model could not form correctly is another. Low self-reported confidence is weaker and worth measuring before trusting. A classifier deciding complexity up front avoids the double call entirely and is worth building once traffic justifies it.

The arithmetic has to work. Escalating means paying for both calls, so if the cheap model succeeds only half the time and the strong model costs ten times more, you are paying 5.5 times the cheap cost rather than 10, which is a saving, and if the cheap model succeeds a tenth of the time you are paying more than going straight to the strong model. Measure your escalation rate before assuming routing saves money.

Route by task, not by user. A simple classification is cheap for everyone, and a complex extraction is expensive for everyone.

Two costs to remember. Escalation doubles latency for the escalated fraction, which the streaming work in Module 8 makes visible to users. And each model has its own cache, so routing across models means neither accumulates cache hits at the rate a single model would.

[IMAGE PROMPT M10-6
Purpose: Show cost-aware routing with escalation and make clear that escalation costs both calls.
Visual type: Flow diagram with cost annotations on each path.
Prompt: A clean educational flow diagram reading left to right. An entry box labelled "request" leads to a decision diamond labelled "obviously simple?". Its "yes" branch leads directly to a terminal labelled "cheap model" annotated "cost: 1x". Its "no" branch leads to a box labelled "cheap model attempt", which leads to a second decision diamond labelled "output valid and acceptable?". That diamond's "yes" branch leads to a terminal labelled "return cheap result" annotated "cost: 1x". Its "no" branch leads to a box labelled "strong model", ending at a terminal labelled "return strong result" annotated "cost: 1x + 10x, both calls billed". Beneath the diagram sits a small worked example box containing three lines: "escalation rate 50%: average 6.5x", "escalation rate 10%: average 2.0x", "escalation rate 90%: average 10.0x, worse than going direct".
Required elements: Two decision diamonds with their questions, three terminal outcomes with cost annotations, explicit note that both calls are billed on escalation, the worked example box with three escalation rates.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, diamonds for decisions.
Layout: Horizontal flow reading left to right with branches, worked example box beneath.
Text labels: "request", "obviously simple?", "cheap model", "cheap model attempt", "output valid and acceptable?", "return cheap result", "strong model", "return strong result", "cost: 1x", "cost: 1x + 10x, both calls billed", "escalation rate 50%: average 6.5x", "escalation rate 10%: average 2.0x", "escalation rate 90%: average 10.0x, worse than going direct", "yes", "no".
Aspect ratio: 16:9
Accessibility: Label every branch yes or no in text and state all cost multipliers in words rather than encoding them visually.
Avoid: Vendor logos, real model names, decorative elements, tiny text, watermarks.
Alt text: Routing flow showing simple requests going straight to a cheap model, others attempting the cheap model first and escalating to a strong model when output is unacceptable, with a worked example showing average cost at three escalation rates.
END IMAGE PROMPT]

Batch and asynchronous pricing tiers

Providers commonly offer an asynchronous batch mode at around half price, with results returned within a window that is often up to twenty-four hours. [VOLATILE: discount levels and turnaround windows differ by provider. Verify current terms.]

What suits batch. Corpus-wide embedding and extraction, which is most of what recipe-extractor does offline. Evaluation runs. Backfills after a prompt change. Enrichment and classification jobs. Anything where nobody is waiting.

What does not. Anything in a user's request path.

The architectural consequence is worth stating: separating latency-sensitive from latency-tolerant work is a cost decision as much as a design one. A pipeline that processes new documents in real time because that is how it was written, when nobody reads them for hours, is paying double for nothing.

Batch and caching sometimes combine, which compounds the saving on repeated prefixes, though the stacking rules vary by provider.

Note also that the checkpointing and resumability from Module 4 is what makes batch practical: a job submitted now and collected tomorrow needs to survive your process restarting in between.

Trading output tokens for latency and price

Output is the expensive half and it is also the slow half, so reducing it improves both at once.

Ask for less. "Answer in two sentences" is a cost control. So is a structured output schema with a small number of fields, since it bounds what the model produces.

Set max_tokens deliberately. It is a ceiling on the worst case, and leaving it at a large default means an occasional runaway response costs many times a normal one.

Avoid asking for restated input. A response that echoes the question before answering is paying output rates for tokens you already sent at input rates.

Reduce reasoning effort where the task does not need it. Extraction from a clearly structured document rarely benefits from extended reasoning, and the setting is a direct multiplier on output cost.

Do not compress at the expense of correctness. A shorter answer that omits a needed caveat is not a saving, and the repair loop from Module 6 costs more than the tokens you avoided. The rule is to remove tokens that carry no information, such as preambles, restatements, and apologies, and to leave the ones that do.

Cost regression tests in CI

A prompt change that doubles spend looks exactly like a prompt change that does not. Catch it before it ships.

COST_CEILINGS = {
    "extraction": Decimal("0.004"),
    "chat_turn": Decimal("0.012"),
    "summarize": Decimal("0.002"),
}


async def test_extraction_cost_within_ceiling(fixture_documents, recording_provider):
    """Fail the build if average cost per extraction exceeds its ceiling."""
    costs = []
    for document in fixture_documents:
        async with attributed("extraction"):
            result = await extract_with_repair(document, schema=ExtractedRecipe)
        costs.append(recording_provider.last_cost())

    average = sum(costs) / len(costs)
    assert average <= COST_CEILINGS["extraction"], (
        f"extraction now averages {average}, ceiling is {COST_CEILINGS['extraction']}"
    )

Three things make this work in practice.

Use recorded responses rather than live calls, so the test is deterministic, free, and runnable on every commit. The recording carries the token counts, which is all the cost calculation needs. This is the cassette approach the testing module covers.

Track tokens rather than money as the primary assertion. Prices change and your token consumption is what you control, so a token ceiling fails for the right reason while a dollar ceiling can fail because a provider raised prices.

Report the trend, not just the pass or fail. A change taking cost from 0.0021 to 0.0038 passes a ceiling of 0.004 and is a warning sign. Printing the delta against the previous run turns a binary gate into information.

Add a ceiling for context size too, since a retrieval change that quietly doubles the chunks sent will show up as cost long before anyone notices it as quality.

Concept check. Your chat feature has a 4,000 token system prompt with tool definitions, and a cache hit rate of two percent. Conversations average eight turns. What is the likely cause and what is the first thing to check?

Answer

Something volatile is sitting before or inside the stable prefix, so the byte-identical match fails on every request after the first.

The first thing to check is whether anything varying is being placed early: a timestamp or session identifier in the system prompt, a user name or personalisation string, or tool definitions serialised from a structure whose ordering is not stable. Any of those invalidates the cache from that point on, so you pay the write premium repeatedly and collect no reads.

Eight-turn conversations with a 4,000 token stable prefix is close to an ideal caching case, so a two percent hit rate is a bug rather than a limitation. Once fixed, the prefix is written once per conversation and read seven times, which is comfortably past the break-even point.

Worth checking second: whether the prefix is above the provider's minimum size for caching to engage, and whether requests within a conversation are far enough apart in time that the cache has expired between turns.