Lesson 1: Counting

The problem this lesson solves

The invoice arrives and it is twelve thousand dollars. Nobody can say which feature caused it.

Meanwhile the pipeline fails intermittently on long documents, because conversation history plus retrieved chunks grew past the context window and the request was rejected. And the estimate that said this would cost four thousand dollars was built on dividing character counts by four, which was wrong by thirty percent because a third of the corpus is not in English.

Three separate failures, one root cause: the system does not know how many tokens it is sending. Everything in this module depends on fixing that first.

Cost is not a finance problem to be reviewed monthly. It is an engineering problem with the same shape as memory or latency: a resource consumed per request, which can be measured, budgeted, and bounded in code.

How tokenization actually works

A token is the unit a model reads. It is not a character and not a word. Tokenizers split text into subword pieces chosen so that common sequences become single tokens and rare ones become several.

"unbelievable"   ->  ["un", "bel", "iev", "able"]      4 tokens
"the"            ->  ["the"]                            1 token
"antidisestablishmentarianism"  ->  many tokens

The consequences follow directly from that.

Common English words are one token. The tokenizer's vocabulary was built from text where they appear constantly, so they earned a slot.

Rare words, names, and identifiers fragment. A product code such as RCP-2847-B may be five or six tokens, because no part of it was common enough to earn one.

Whitespace is usually attached to the following word, so " the" and "the" are different tokens. This is why counting tokens by splitting on spaces gives the wrong answer even for English.

Numbers fragment unpredictably. 2847 might be one token or three depending on the tokenizer, which is part of why models are unreliable at arithmetic.

Non-English text costs more. Text in a script the tokenizer saw less of during training fragments into more pieces, so the same meaning in one language can cost two or three times the tokens of another. This is the fertility of the tokenizer for that language, and it is a real cost difference for a multilingual corpus, not a rounding error.

Code fragments heavily. Indentation, brackets, and identifiers all consume tokens, so a file of code costs considerably more than a paragraph of prose of the same character length.

Why character and word counts lie

The rule of thumb is that a token is about four characters, and it is roughly right for English prose and badly wrong for everything else.

[VOLATILE: these ratios are illustrative and vary by tokenizer. Measure your own corpus rather than trusting any table, including this one.]

A pipeline estimating with character division and processing a mixed corpus will be wrong in both directions: under-estimating on code and non-English, over-estimating on clean English. The under-estimates are the dangerous half, because they cause context window overflows and budget breaches rather than merely conservative behaviour.

Estimation has one legitimate use. A rough count is fine for deciding whether something is obviously far under a limit. It is not fine for a pre-flight check, a budget guard, or a cost projection, which are exactly the three places people use it.

[IMAGE PROMPT M10-1
Purpose: Show why a fixed characters-per-token ratio fails across content types, and that the error is largest where it matters most.
Visual type: Comparison chart with paired bars and an error annotation.
Prompt: A clean educational chart with five rows, each labelled on the left: "English prose", "Technical English", "Python code", "JSON payload", "Non-Latin script". Each row shows two horizontal bars of different lengths side by side, the upper one labelled "estimated: characters divided by 4" drawn identically across all five rows to show a fixed estimate, and the lower one labelled "actual token count" drawn at differing lengths, shortest for English prose and longest for the non-Latin script row. A bracket to the right of each row marks the gap between the two bars, annotated "over-estimate" for the English prose row and "under-estimate" for the code, JSON, and non-Latin rows, with the largest bracket on the non-Latin row. A note beneath reads "under-estimates cause context overflows and budget breaches, over-estimates only waste headroom".
Required elements: Five labelled content types, a constant-length estimate bar and a variable-length actual bar in each row, gap brackets with over-estimate and under-estimate labels, the explanatory note.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, consistent bar heights.
Layout: Horizontal bars in five stacked rows reading left to right, labels on the left, gap brackets on the right.
Text labels: "English prose", "Technical English", "Python code", "JSON payload", "Non-Latin script", "estimated: characters divided by 4", "actual token count", "over-estimate", "under-estimate", "under-estimates cause context overflows and budget breaches, over-estimates only waste headroom".
Aspect ratio: 16:9
Accessibility: Distinguish estimate from actual by bar position and text label rather than colour alone, and label every gap in words.
Avoid: Precise numeric axis values, vendor logos, decorative elements, tiny text, watermarks.
Alt text: Chart comparing a fixed characters-divided-by-four estimate against actual token counts for five content types, showing over-estimation for English prose and progressively larger under-estimation for technical text, code, JSON, and non-Latin scripts.
END IMAGE PROMPT]

Counting before you send

Count with the actual tokenizer. For models using a published tokenizer, tiktoken does it locally with no network call.

import tiktoken

_encoding = tiktoken.get_encoding("cl100k_base")


def count_text_tokens(text: str) -> int:
    return len(_encoding.encode(text))

[VOLATILE: encoding names change with model families, and not every provider publishes a local tokenizer. Verify which applies to the models you use.]

Two practical notes. Load the encoding once at module level rather than per call, since construction is expensive relative to encoding. And lru_cache from Module 3 is appropriate here when the same text is counted repeatedly, keyed on the text and the encoding name.

Messages cost more than their content. A chat request is not a concatenation of message strings. Each message carries structural tokens for its role and boundaries, and the request as a whole has a small overhead.

MESSAGE_OVERHEAD_TOKENS = 4      # per message, approximate
REQUEST_OVERHEAD_TOKENS = 3      # per request, approximate


def count_message_tokens(messages: list[Message]) -> int:
    """Count a chat request including per-message structural overhead."""
    total = REQUEST_OVERHEAD_TOKENS
    for message in messages:
        total += MESSAGE_OVERHEAD_TOKENS
        total += count_text_tokens(message.role)
        total += count_text_tokens(message.content)
    return total

[VOLATILE: the overhead values differ by provider and model family. Verify against a real request rather than trusting constants.]

Tools and schemas count too. Tool definitions are serialised into the request, so a set of ten tools with detailed descriptions can be a thousand tokens on every call. The same is true of the JSON schema for structured output from Module 6. Both are easy to forget precisely because you did not write them into the prompt yourself.

Verify against the provider's reported usage. Every response reports actual token counts. Compare yours against theirs on a sample and record the discrepancy.

def check_estimate(estimated: int, reported: int, *, request_id: str) -> None:
    """Log when our count drifts from the provider's, so we notice."""
    if reported == 0:
        return
    error = abs(estimated - reported) / reported
    if error > 0.05:
        logger.warning(
            "token_estimate_drift",
            extra={"request_id": request_id, "estimated": estimated,
                   "reported": reported, "error_pct": round(error * 100, 1)},
        )

If your counter drifts, everything built on it drifts too, and this check is how you find out before the invoice does.

Tokenizer differences across providers

Token counts do not transfer between providers. The same text tokenized by two different model families produces different counts, sometimes differing by ten to twenty percent.

Three consequences.

A cost comparison between providers must count with each provider's own tokenizer. Comparing at one count and two prices is arithmetic on a wrong number.

A context budget computed for one model is wrong for another, and the direction matters: switching to a model whose tokenizer is less efficient for your content can push requests over a limit that previously fit.

Provider-specific counting belongs in the adapter from Module 7, alongside the other provider differences it already absorbs.

class LLMProvider(Protocol):
    name: str
    capabilities: Capabilities

    def count_tokens(self, messages: list[Message], *, model: str) -> int: ...

Some providers offer a counting endpoint rather than a local tokenizer, which costs a network call. Cache aggressively when so, since the same system prompt is counted on every request.

Counting images, documents, and audio

Non-text inputs consume tokens too, usually far more than people expect.

Images are charged by a formula based on dimensions, typically tiling the image and charging per tile plus a base amount. The practical points: a large image can cost more than a page of text, resizing before sending reduces cost as well as payload size, and the Module 4 rule about downscaling has a cost justification as well as a limits justification.

def estimate_image_tokens(width: int, height: int) -> int:
    """Approximate image token cost by tiling. Provider specific."""
    tiles = math.ceil(width / TILE_SIZE) * math.ceil(height / TILE_SIZE)
    return BASE_IMAGE_TOKENS + tiles * TOKENS_PER_TILE

[VOLATILE: tile sizes, base costs, and formulas differ by provider and change. Verify each one.]

Documents sent as PDFs are usually converted to text, images, or both before the model sees them, so a 40-page PDF can be tens of thousands of tokens. Estimate before sending rather than discovering the cost afterwards.

Audio is charged by duration or by converted tokens depending on the provider and the mode.

The rule that covers all three: anything you send costs tokens, and your counter must know about every input type your system accepts. A counter that handles text and silently ignores images will under-report on exactly the requests that cost the most.

A count_tokens utility the whole codebase trusts

One counter, used everywhere, is the foundation for the rest of this module.

@dataclass(frozen=True)
class TokenCount:
    text: int
    images: int
    tools: int
    total: int
    model: str
    is_estimate: bool


class TokenCounter:
    """The single source of truth for how many tokens a request will cost."""

    def __init__(self, provider: LLMProvider, model: str) -> None:
        self._provider = provider
        self._model = model

    def count_request(
        self,
        messages: list[Message],
        *,
        tools: list[dict] | None = None,
        images: list[ImageSpec] | None = None,
    ) -> TokenCount:
        text = self._provider.count_tokens(messages, model=self._model)
        tool_tokens = self._count_tools(tools or [])
        image_tokens = sum(estimate_image_tokens(i.width, i.height) for i in images or [])
        return TokenCount(
            text=text,
            images=image_tokens,
            tools=tool_tokens,
            total=text + tool_tokens + image_tokens,
            model=self._model,
            is_estimate=image_tokens > 0,
        )

Three properties make it trustworthy.

It is provider aware, delegating to the adapter so the count matches the model actually being called.

It counts everything, including tools and images, which are the two categories most often omitted.

It reports whether the number is exact, since image token costs are formula-based estimates. The is_estimate flag carries forward into cost records, exactly as StreamUsage did in Module 8, so a reconciliation can distinguish measured figures from inferred ones.

Concept check. Your token estimate matches the provider's reported count almost exactly in development and is consistently ten percent low in production. What is the most likely cause?

Answer

Something is in the production request that is not in your count. The two most common candidates are tool definitions and a structured output schema, both of which are serialised into the request by your client library rather than written by you, and both of which are easy to omit from a counter that only walks the message list.

Retrieved context is a third candidate if the counting happens before assembly rather than after. Count the request you are actually about to send, not the pieces you assembled it from.

The reason it appears only in production is usually that development runs a simpler path: fewer tools registered, no structured output, or shorter retrieval results. The drift check comparing your estimate against the reported usage is what surfaces this, which is why it belongs in the code rather than in a one-off investigation.