Lesson 6: Correctness Under Retry

Idempotency keys

Module 4 established the principle: prefer duplicating work over losing data, then make writes idempotent so duplication is harmless. Retries create the same problem at the level of external calls, and it is sharper there because the duplicate may cost money or have a real-world effect.

Consider a request that times out. Your client saw no response, so you retry. But the timeout does not tell you whether the server processed it. There are three possibilities: the request never arrived, it arrived and failed, or it arrived and succeeded, and only the response was lost. In the third case, retrying performs the operation twice.

For a read this is harmless. For a generation call it costs twice. For anything that creates a record or charges an account, it is a correctness bug.

An idempotency key is a unique identifier you attach to a request. The server records it, and if the same key arrives again, it returns the original result instead of performing the operation a second time.

import uuid


def embed_batch(texts: list[str], *, idempotency_key: str | None = None) -> list[list[float]]:
    """Embed a batch. The same key is safe to send more than once."""
    key = idempotency_key or str(uuid.uuid4())
    return call_provider(
        {"input": texts},
        headers={"Idempotency-Key": key},
    )

The critical detail is where the key is generated. It must be created once, before the first attempt, and reused across every retry of that same logical operation. Generating it inside the retry loop produces a new key per attempt, which defeats the entire mechanism while looking correct.

# Wrong: a new key on every attempt, so the server sees distinct requests
@with_retry()
def embed_batch(texts: list[str]) -> list[list[float]]:
    return call_provider({"input": texts}, headers={"Idempotency-Key": str(uuid.uuid4())})


# Right: the key is fixed before retries begin
def embed_document(doc: Document) -> list[list[float]]:
    key = f"embed:{doc.content_hash}"
    return _embed_with_retry(doc.chunks, idempotency_key=key)

Deriving the key from content is better than a random value, because it makes the operation idempotent across runs as well as across retries. If your job is killed and resumed, a content-derived key means the resumed attempt is recognised as the same operation. A random key would not be, since it lives only in the memory of a process that no longer exists.

When the provider does not support idempotency keys, which is common, you must handle it yourself. The options are to make the operation naturally repeatable, such as writing to a keyed store rather than appending, or to record locally what you have already requested, which is what the checkpoint from Module 4 does.

Defensive programming at trust boundaries

Module 2 established parse at the boundary and trust inside. Reliability adds a second dimension to the same principle: at a trust boundary, assume not just that the data may be malformed, but that the other side may be slow, absent, or wrong.

Your trust boundaries are the places where something you do not control meets something you do: files from a scraper, HTTP requests arriving at your service, responses from a provider, rows from a database, messages from a queue, and configuration from the environment.

At each one, five defences apply.

Validate the shape. Parse into a typed record and reject what does not fit. This is Module 2, unchanged.

Bound everything. Set a timeout, a size limit, and a count limit on anything crossing the boundary. A response with no size limit can exhaust memory as surely as an image with no pixel limit.

MAX_RESPONSE_BYTES = 10 * 1024 * 1024

if len(response.content) > MAX_RESPONSE_BYTES:
    raise ProviderError("response exceeded size limit")

Do not trust semantics you did not verify. A 200 status means the request was accepted, not that the content is right. A model returning valid JSON has not thereby returned true or complete JSON. This is the model error category from Lesson 1, and the only defence is checking the content.

Fail loudly at the boundary and quietly inside. A validation failure at the edge should raise a clear, specific error naming what was wrong. Inside, where data has already been checked, functions should not be re-checking and silently returning None.

Log enough to diagnose, and nothing sensitive. Record the failure type, the identifier of the affected record, and the status code. Do not record the payload, the API key, or the full model response, following the discipline established in Module 3.

def parse_embedding_response(payload: dict, *, expected_count: int) -> list[list[float]]:
    """Validate a provider response before letting it into the pipeline."""
    data = payload.get("data")
    if not isinstance(data, list):
        raise OutputError("response missing 'data' array")

    if len(data) != expected_count:
        raise OutputError(
            f"expected {expected_count} embeddings, received {len(data)}"
        )

    vectors: list[list[float]] = []
    for index, item in enumerate(data):
        vector = item.get("embedding") if isinstance(item, dict) else None
        if not isinstance(vector, list) or not vector:
            raise OutputError(f"embedding {index} missing or empty")
        vectors.append([float(value) for value in vector])

    return vectors

The count check is the one people omit, and it is the one that catches the genuinely dangerous failure. A provider returning nine embeddings for ten inputs, with no error, silently misaligns every chunk with the wrong vector. Nothing downstream will detect it, the pipeline will complete successfully, and the corpus will be quietly wrong. That is a logical error from Lesson 1, created by trusting a boundary, and one line of validation prevents it.

Predict the output.

@with_retry(max_attempts=3)
def charge_and_embed(doc: Document) -> list[float]:
    key = str(uuid.uuid4())
    return call_provider({"input": doc.content}, headers={"Idempotency-Key": key})

The first attempt reaches the provider, is processed successfully, and the response is lost to a timeout. The retry succeeds normally. How many times was the document processed and billed?

Answer

Twice. The idempotency key is generated inside the decorated function, so each attempt produces a different key and the provider treats the retry as an entirely new request. The first call succeeded on the server even though your client never saw the response, and the second call did the work again.

The fix is to generate the key outside the retried function and pass it in, ideally derived from the document content hash so that it is stable across process restarts as well as across retries. The subtlety is that the code looks correct: the key is present, the header is set, and nothing raises. Only the placement is wrong.