Lesson 3: Cost Accounting

Input, output, and cached-input asymmetry

Tokens are not priced uniformly, and the differences are large enough to change design decisions.

Output costs several times input. A common ratio is four to five times, and the reason is mechanical: input tokens are processed in parallel during prefill while output tokens are generated one at a time.

The consequence is that output length is usually the dominant cost lever. Halving your prompt saves less than halving your response.

Cached input costs a fraction of standard input. Where a provider supports prompt caching, a cache read typically costs around a tenth of the standard input rate, and a cache write costs somewhat more than standard, commonly around 1.25 times. [VOLATILE: cache read discounts range from roughly 50 to 90 percent depending on provider and model, and write premiums vary. Verify current figures.]

The break-even follows directly: paying a write premium once and reading twice is usually where it starts paying, and every read after that is nearly free. Lesson 4 covers how to structure prompts so the reads actually happen.

Some providers price long contexts differently, applying a higher rate above a threshold, and on some the higher rate applies to the whole request rather than to the excess. A request just over that line can cost substantially more than one just under it, which is worth knowing before you tune top_k upward.

Tools may bill separately. A built-in web search or file search tool can carry a per-call charge on top of the tokens for the content it returns. A cost model counting only tokens will under-report for any system using them.

[IMAGE PROMPT M10-4
Purpose: Show the relative cost of input, cached input, output, and reasoning tokens, and that reasoning tokens are billed but invisible.
Visual type: Comparative bar chart with an annotated invisible component.
Prompt: A clean educational bar chart with four horizontal bars of differing lengths, labelled from top to bottom: "cached input read" drawn very short, "standard input" drawn short, "cache write" drawn slightly longer than standard input, and "output" drawn roughly four to five times the length of standard input. Beneath these, a fifth bar labelled "reasoning tokens" is drawn the same length as the output bar but with a dashed or hatched outline, annotated "billed as output, never shown to the user". A relative-scale note sits at the left reading "relative cost per token, not to scale with any specific provider". A callout to the right of the output bar reads "output is usually the dominant cost lever". A second callout beside the cache write bar reads "pays for itself after about two reads".
Required elements: Four solid bars in the stated order with correct relative lengths, a fifth dashed reasoning bar, the relative-scale note, two callouts.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, no numeric axis.
Layout: Horizontal bars stacked vertically with labels on the left and callouts on the right.
Text labels: "cached input read", "standard input", "cache write", "output", "reasoning tokens", "billed as output, never shown to the user", "relative cost per token, not to scale with any specific provider", "output is usually the dominant cost lever", "pays for itself after about two reads".
Aspect ratio: 16:9
Accessibility: Distinguish the reasoning bar by outline style and its text label rather than colour, and state explicitly that no absolute scale is implied.
Avoid: Specific prices, vendor names, decorative elements, tiny text, watermarks.
Alt text: Bar chart comparing relative per-token costs, showing cached input reads cheapest, standard input and cache writes moderate, output several times higher, and reasoning tokens billed at output rates while remaining invisible to the user.
END IMAGE PROMPT]

Reasoning tokens and invisible output

Reasoning models generate internal tokens before their visible answer. Those tokens are billed as output and consume context window, and you never see them.

This produces a distinctive failure: a feature whose visible responses are two hundred tokens costs as though they were two thousand, and nothing in the response explains why.

@dataclass(frozen=True)
class Usage:
    input_tokens: int
    cached_input_tokens: int
    output_tokens: int
    reasoning_tokens: int          # included in output_tokens on most providers

[VOLATILE: whether reasoning tokens are reported separately, and whether they are included in the output total or additional to it, differs by provider. Verify before computing costs, since double counting or omitting them are both easy.]

Three practical points. Reasoning effort is usually configurable, and lowering it is a direct cost lever for tasks that do not need deep reasoning. The output reserve from Lesson 2 must accommodate reasoning tokens, or a reasoning model will run out of window mid-thought. And when comparing a reasoning model against a standard one, compare total cost per completed task rather than per visible token, since the reasoning model may need fewer attempts.

Cost per request, session, user, and feature

A total spend figure is not actionable. Four levels of granularity, each answering a different question.

@dataclass(frozen=True)
class CostRecord:
    request_id: str
    timestamp: datetime

    # Attribution
    feature: str                   # "extraction", "chat", "summarize"
    user_id: str | None
    session_id: str | None

    # Measurement
    provider: str
    model: str
    usage: Usage
    cost_usd: Decimal
    is_estimated: bool

    # Context
    latency_ms: int
    was_cached: bool
    outcome: Literal["success", "error", "truncated", "cancelled"]

Note Decimal rather than float for money. Floating point accumulation error across millions of records is real, and money is exactly the case where it is not acceptable.

Per request answers whether one call was unusually expensive.

Per session answers what a whole conversation cost, which is the unit that matters for a chat product, since a cheap request repeated forty times is not cheap.

Per user is what quotas and abuse detection need.

Per feature is what product decisions need, and it is the level most often missing.

Attribution

Attribution is tagging spend so a number maps to a decision. Without it, "the bill is twelve thousand dollars" leads nowhere. With it, "extraction is nine thousand of the twelve, and eighty percent of that is retries on documents that fail validation" points at something specific to fix.

The mechanism is that every call carries its context. contextvars from Python's standard library propagates it across async boundaries without threading a parameter through every function.

from contextvars import ContextVar

current_feature: ContextVar[str] = ContextVar("current_feature", default="unknown")
current_user: ContextVar[str | None] = ContextVar("current_user", default=None)


@asynccontextmanager
async def attributed(feature: str, *, user_id: str | None = None):
    """Tag every model call inside this block with a feature and user."""
    feature_token = current_feature.set(feature)
    user_token = current_user.set(user_id)
    try:
        yield
    finally:
        current_feature.reset(feature_token)
        current_user.reset(user_token)

async with attributed("extraction", user_id=user.id):
    result = await extract_with_repair(document, schema=ExtractedRecipe)

Every provider call inside that block records the feature and user without any function signature changing. The finally restores the previous value, which matters for nested contexts.

Tag at a useful granularity. "Extraction" is a feature. "Extraction repair attempt" is a sub-feature worth separating, because knowing that a third of extraction spend is repairs is the kind of finding that changes what you work on.

Record failures too. A request that failed validation still cost money. Attribution that only counts successes understates spend by exactly the amount you most want to reduce.

A pricing table that survives price changes

Hardcoded prices go stale silently and are usually scattered.

@dataclass(frozen=True)
class ModelPricing:
    model: str
    input_per_million: Decimal
    output_per_million: Decimal
    cached_input_per_million: Decimal | None
    cache_write_per_million: Decimal | None
    effective_from: date
    effective_until: date | None = None


class PricingTable:
    """Prices with effective dates, so historical costs stay correct."""

    def __init__(self, entries: list[ModelPricing]) -> None:
        self._entries = entries

    def price_for(self, model: str, when: date) -> ModelPricing:
        matches = [
            e for e in self._entries
            if e.model == model
            and e.effective_from <= when
            and (e.effective_until is None or when <= e.effective_until)
        ]
        if not matches:
            raise ConfigurationError(f"no pricing for {model} on {when}")
        return max(matches, key=lambda e: e.effective_from)

    def cost(self, usage: Usage, *, model: str, when: date) -> Decimal:
        p = self._entries and self.price_for(model, when)
        uncached = usage.input_tokens - usage.cached_input_tokens
        total = (
            Decimal(uncached) * p.input_per_million
            + Decimal(usage.output_tokens) * p.output_per_million
        ) / Decimal(1_000_000)
        if usage.cached_input_tokens and p.cached_input_per_million:
            total += (
                Decimal(usage.cached_input_tokens) * p.cached_input_per_million
            ) / Decimal(1_000_000)
        return total

Four properties that make it survive.

Effective dates mean a price change does not rewrite history. Last quarter's costs stay correct.

Data, not code. Load it from a configuration file so updating a price is a config change rather than a deployment.

An unknown model raises. Defaulting to zero or to some other model's price produces a cost report that is quietly wrong, which is worse than an error.

Cached tokens are priced separately, and note that they are subtracted from the input total rather than added, since providers report cached tokens as part of input rather than in addition to it. Getting this backwards double counts.

Reconciling against the provider invoice

Your numbers and the provider's will differ. Reconcile monthly and investigate gaps.

Common causes, and all of them are things worth knowing about your own system.

Requests you did not record. Retries, health checks, and calls from a path that lacks attribution. This is the most common cause and the most useful finding.

Failed requests that still billed. A request that timed out on your side may have completed on theirs, which Module 5's idempotency discussion covered.

Cancelled streams. Module 8 noted that a client disconnect may leave usage unrecorded while the tokens were still generated.

Estimates counted as measurements. Image tokens and any usage inferred rather than reported, which is why the is_estimated flag exists.

Stale prices. Your table says one thing and the provider changed it.

Tool or feature charges billed outside token pricing.

def reconcile(period: DateRange, invoice_total: Decimal) -> ReconciliationReport:
    """Compare recorded spend against the invoice and quantify the gap."""
    recorded = sum_recorded_cost(period)
    gap = invoice_total - recorded
    return ReconciliationReport(
        period=period,
        recorded=recorded,
        invoiced=invoice_total,
        gap=gap,
        gap_percent=float(gap / invoice_total * 100) if invoice_total else 0.0,
        estimated_portion=sum_estimated_cost(period),
        requests_recorded=count_records(period),
    )

A gap under a few percent is normal. A gap of thirty percent means a whole category of requests is unattributed, and finding it is usually a day's work with a large payoff.

Note that provider dashboards typically report usage a day or more in arrears, so same-day comparison will always look wrong. Reconcile on closed periods.