Part 27 · Error Handling and Reliability Engineering

Lesson 3: Retry Logic

8 min read·9 Sept 2026

Retryable versus non-retryable

The single most important reliability decision is whether a failure will succeed on a second attempt. Retrying something that cannot succeed wastes time, spends money, and adds load to a service that may already be struggling.

Retryable, meaning the same request may succeed later:

Network errors: connection reset, DNS failure, connection refused, timeout
429, rate limited
500, an unspecified server error
502, 503, 504, meaning bad gateway, service unavailable, and gateway timeout
Provider-specific overload responses

Not retryable, meaning the same request will fail identically:

400, malformed request. Your payload is wrong and will be wrong again.
401 and 403, authentication and authorisation. Your key will not become valid.
404, not found.
422, semantically invalid request.
Any ValidationError from your own code
Any runtime error in your own code

The ambiguous cases, which need a decision rather than a default:

A 408 request timeout is usually retryable. A 409 conflict depends entirely on the API, and often means retrying is exactly wrong. A 413 payload too large is not retryable as-is, though it may be retryable after splitting the payload, which is a different request. A model error such as malformed JSON is retryable in a special sense: the same request may produce different output, because generation is not deterministic, so retrying can work where it would not for a 400.

Encode the decision once rather than scattering status codes through your code:

python

RETRYABLE_STATUS = frozenset({408, 429, 500, 502, 503, 504})


def is_retryable(exc: Exception) -> bool:
    """Decide whether this failure might succeed on another attempt."""
    if isinstance(exc, (RateLimitError, ProviderTimeoutError, ProviderOverloadedError)):
        return True
    if isinstance(exc, httpx.TransportError):
        return True
    if isinstance(exc, ProviderError):
        return False                    # other provider errors are permanent
    return False                        # everything else, including our own bugs

Note the default. Anything not explicitly known to be retryable is treated as permanent, which fails safely: an unnecessary failure is visible and cheap, while an infinite retry loop against a permanent error is expensive and can go unnoticed for hours.

Exponential backoff and jitter

Retrying immediately is nearly always wrong. If a service is overloaded, an instant retry adds load at the worst moment.

Exponential backoff doubles the wait between attempts.

python

import time


def retry_with_backoff(func, *, max_attempts: int = 5, base_delay: float = 1.0):
    for attempt in range(max_attempts):
        try:
            return func()
        except Exception as exc:
            if not is_retryable(exc) or attempt == max_attempts - 1:
                raise
            delay = base_delay * (2 ** attempt)      # 1, 2, 4, 8 seconds
            time.sleep(delay)

Two details in the guard clause. It re-raises immediately for non-retryable errors, so a 400 fails in milliseconds rather than after four pointless waits. And it re-raises on the final attempt rather than sleeping and then giving up, which would waste the last delay.

Jitter adds randomness to the delay, and it is not a refinement. It is what makes backoff work at all when more than one client is involved.

Consider the failure it prevents. A provider goes down for thirty seconds. Two hundred of your workers all fail at the same moment. All two hundred wait exactly one second, then all two hundred retry simultaneously. The provider, just recovering, is hit by two hundred requests at once and fails again. All two hundred wait exactly two seconds, and it happens again. This is the thundering herd, and pure exponential backoff makes it worse by synchronising the clients rather than spreading them.

Jitter breaks the synchronisation:

python

import random

# Full jitter: sleep for a random duration up to the computed delay
delay = random.uniform(0, base_delay * (2 ** attempt))

# Equal jitter: half fixed, half random
computed = base_delay * (2 ** attempt)
delay = computed / 2 + random.uniform(0, computed / 2)

Full jitter spreads clients most effectively and is the usual recommendation. Equal jitter guarantees some minimum wait, which is occasionally preferable when the base delay is meaningful.

[IMAGE PROMPT M5-3
Purpose: Show how synchronised retries create load spikes and how jitter spreads them out.
Visual type: Two-panel timeline comparison showing request arrival density.
Prompt: A clean educational comparison with two stacked horizontal timelines sharing the same time axis marked at 0s, 1s, 2s, 4s, and 8s. The upper timeline is headed "Backoff without jitter" and shows tall narrow spikes of stacked request marks positioned exactly at 1s, 2s, 4s, and 8s, with all marks aligned in single columns. Each spike is annotated "200 requests at once". A small icon of a server beneath the timeline is marked "fails again". The lower timeline is headed "Backoff with full jitter" and shows the same total number of request marks but scattered across the intervals between 0s and 1s, 0s and 2s, 0s and 4s, and 0s and 8s, forming low even bands rather than spikes. Each band is annotated "spread across the window". The server icon beneath is marked "recovers". A caption beneath both reads "same number of retries, different arrival pattern".
Required elements: Shared time axis with the marked points, tall aligned spikes above and scattered low bands below, request-count annotations, server icons with their outcomes, the shared caption.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, request marks drawn as small identical ticks.
Layout: Two horizontal timelines stacked vertically and aligned to a shared axis, caption centred beneath.
Text labels: "Backoff without jitter", "Backoff with full jitter", "0s", "1s", "2s", "4s", "8s", "200 requests at once", "spread across the window", "fails again", "recovers", "same number of retries, different arrival pattern".
Aspect ratio: 16:9
Accessibility: Convey the difference through the density and position of tick marks plus explicit labels rather than colour alone.
Avoid: Vendor logos, decorative elements, tiny text, watermarks, clutter.
Alt text: Two timelines showing that exponential backoff without jitter causes all retries to arrive in synchronised spikes that keep a recovering server failing, while full jitter scatters the same retries across each window and lets the server recover.
END IMAGE PROMPT]

Respecting Retry-After

When a provider returns 429 or 503, it often includes a Retry-After header telling you exactly how long to wait. That is better information than any backoff calculation you can make, because it comes from the service that knows.

The header has two formats, and handling only one is a common bug:

python

from email.utils import parsedate_to_datetime
from datetime import datetime, timezone


def parse_retry_after(value: str | None) -> float | None:
    """Parse a Retry-After header, which may be seconds or an HTTP date."""
    if not value:
        return None

    value = value.strip()
    if value.isdigit():
        return float(value)

    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None

    delta = (when - datetime.now(timezone.utc)).total_seconds()
    return max(0.0, delta)

Then prefer it over your computed delay:

python

def compute_delay(exc: Exception, attempt: int, base_delay: float) -> float:
    """Use the provider's instruction when given, otherwise backoff with jitter."""
    retry_after = getattr(exc, "retry_after", None)
    if retry_after is not None:
        return min(retry_after, MAX_DELAY)
    return random.uniform(0, base_delay * (2 ** attempt))

Note the cap. A provider can return a Retry-After of hours, and blindly sleeping for it stalls your job. Cap it, and if the cap is hit, treat that as a signal to stop rather than to wait.

Putting it together as a decorator. This is the mechanism from Module 3 applied to a cross-cutting concern, which is exactly what decorators are for.

python

import functools
import random
import time
from collections.abc import Callable
from typing import Any, TypeVar

T = TypeVar("T")
MAX_DELAY = 60.0


def with_retry(
    *,
    max_attempts: int = 5,
    base_delay: float = 1.0,
) -> Callable[[Callable[..., T]], Callable[..., T]]:
    """Retry retryable failures with jittered exponential backoff."""
    def decorator(func: Callable[..., T]) -> Callable[..., T]:
        @functools.wraps(func)
        def wrapper(*args: Any, **kwargs: Any) -> T:
            for attempt in range(max_attempts):
                try:
                    return func(*args, **kwargs)
                except Exception as exc:
                    last_attempt = attempt == max_attempts - 1
                    if not is_retryable(exc) or last_attempt:
                        raise
                    delay = compute_delay(exc, attempt, base_delay)
                    logger.warning(
                        "retrying %s after %.1fs (attempt %d/%d): %s",
                        func.__name__, delay, attempt + 1, max_attempts, type(exc).__name__,
                    )
                    time.sleep(delay)
            raise AssertionError("unreachable")
        return wrapper
    return decorator


@with_retry(max_attempts=5)
def embed_batch(texts: list[str]) -> list[list[float]]:
    return call_provider({"input": texts})

functools.wraps is present, as Module 3 established it must be. The log line records the exception type rather than its content, following the same discipline as the log_call decorator, and it logs at warning level so that a run producing thousands of retries is visible rather than silent.

Use a library in production. Writing this from scratch teaches the mechanism, which is why it is here. In real code, tenacity is the established choice and handles cases this version does not, such as total time limits and async functions. [VOLATILE: tenacity remains the standard at time of writing, with stamina as a thinner wrapper over it and several newer combined retry and circuit breaker libraries appearing. Verify the current recommendation before publishing.]

python

from tenacity import retry, retry_if_exception, stop_after_attempt, wait_exponential_jitter


@retry(
    retry=retry_if_exception(is_retryable),
    stop=stop_after_attempt(5),
    wait=wait_exponential_jitter(initial=1, max=60),
)
def embed_batch(texts: list[str]) -> list[list[float]]:
    return call_provider({"input": texts})

The is_retryable function you wrote is still the important part. Libraries provide the mechanics, and the classification remains yours.

Concept check. Your retry decorator wraps a function that returns a 400 because a document contains a character the provider rejects. What happens with the code above, and what would happen if is_retryable returned True by default?

Answer

With the code as written, is_retryable returns False for a general ProviderError, so the exception is re-raised on the first attempt and the document is rejected in milliseconds with a clear reason.

If the default were True, the same request would be sent five times, each preceded by a wait, taking roughly fifteen seconds and producing five identical failures. Across a corpus with even one percent of such documents, that is hours of waiting and twenty thousand pointless requests, and the provider sees a client repeatedly sending requests it has already rejected. Defaulting to not retrying is what makes the failure visible instead of expensive.