Lesson 5: Degradation

Circuit breakers

Retries handle a failure. They handle a sustained outage badly.

If a provider is down for ten minutes, every request retries five times with backoff before failing. Each document takes thirty seconds to fail instead of failing immediately, your workers spend the outage sleeping, and the provider receives a constant stream of requests it cannot serve. You have converted a fast, clear failure into a slow, expensive one.

A circuit breaker stops calling a service that is clearly failing, waits, and then tests carefully before resuming. The name comes from electrical circuits, and the analogy holds well: it trips to protect the system, and it must be reset deliberately rather than continuing to conduct.

It has three states.

Closed is normal operation. Calls pass through, and failures are counted. When failures exceed a threshold, the breaker opens.

Open means calls fail immediately without being attempted. No request is sent, no time is spent, and the failing service receives nothing. After a cooldown, the breaker moves to half open.

Half open allows a limited number of trial calls. If they succeed, the breaker closes and normal operation resumes. If any fails, it opens again and the cooldown restarts.

import time
from dataclasses import dataclass, field
from typing import Literal

BreakerState = Literal["closed", "open", "half_open"]


@dataclass
class CircuitBreaker:
    failure_threshold: int = 5
    cooldown_seconds: float = 30.0
    half_open_successes_required: int = 2

    _state: BreakerState = field(default="closed", init=False)
    _failures: int = field(default=0, init=False)
    _successes: int = field(default=0, init=False)
    _opened_at: float = field(default=0.0, init=False)

    def before_call(self) -> None:
        """Raise immediately if the circuit is open. Call this before attempting."""
        if self._state == "open":
            if time.monotonic() - self._opened_at >= self.cooldown_seconds:
                self._state = "half_open"
                self._successes = 0
            else:
                raise ProviderOverloadedError("circuit open, not attempting call")

    def record_success(self) -> None:
        if self._state == "half_open":
            self._successes += 1
            if self._successes >= self.half_open_successes_required:
                self._state = "closed"
                self._failures = 0
        else:
            self._failures = 0

    def record_failure(self) -> None:
        if self._state == "half_open":
            self._trip()
            return
        self._failures += 1
        if self._failures >= self.failure_threshold:
            self._trip()

    def _trip(self) -> None:
        self._state = "open"
        self._opened_at = time.monotonic()
        logger.error("circuit opened after %d failures", self._failures)

[IMAGE PROMPT M5-5
Purpose: Show the three circuit breaker states and the exact transitions between them.
Visual type: State machine diagram.
Prompt: A clean educational state machine diagram with three labelled circular or rounded state nodes arranged in a triangle. The node at the left is labelled "CLOSED" with the sub-label "calls pass through, failures counted". The node at the right is labelled "OPEN" with the sub-label "calls fail immediately, nothing sent". The node at the bottom centre is labelled "HALF OPEN" with the sub-label "limited trial calls allowed". Directed arrows connect them: from CLOSED to OPEN labelled "failures reach threshold"; from OPEN to HALF OPEN labelled "cooldown elapsed"; from HALF OPEN to CLOSED labelled "trial calls succeed"; from HALF OPEN back to OPEN labelled "any trial call fails, cooldown restarts". A self-loop on CLOSED is labelled "success resets the failure count".
Required elements: Three named state nodes with sub-labels, four directed transition arrows each with its condition label, one self-loop on the closed state.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, clearly directed arrowheads.
Layout: Triangular arrangement with CLOSED upper left, OPEN upper right, HALF OPEN bottom centre, arrows curving between them without crossing.
Text labels: "CLOSED", "OPEN", "HALF OPEN", "calls pass through, failures counted", "calls fail immediately, nothing sent", "limited trial calls allowed", "failures reach threshold", "cooldown elapsed", "trial calls succeed", "any trial call fails, cooldown restarts", "success resets the failure count".
Aspect ratio: 4:3
Accessibility: Label every transition in text and use distinct arrow directions rather than colour to distinguish paths.
Avoid: Electrical imagery, decorative icons, screenshots, tiny text, logos, watermarks.
Alt text: State machine showing a circuit breaker moving from closed to open when failures reach a threshold, from open to half open after a cooldown, from half open back to closed when trial calls succeed, and from half open to open again if any trial call fails.
END IMAGE PROMPT]

Getting the settings right matters more than the code. A threshold that is too low trips on ordinary noise and makes your system less available than the provider it is protecting. A cooldown that is too short reopens the circuit repeatedly. A cooldown that is too long leaves you down after the provider recovers. Start around five failures and thirty seconds, then adjust based on what you observe.

One breaker per dependency, not one globally. If your embedding provider is down, your database should still be reachable. Sharing a breaker across dependencies means one failure disables everything.

In production, use a library. pybreaker is the long-standing choice, and several newer libraries combine retry and circuit breaking in one decorator. As with retries, the value of building it once is understanding the state machine well enough to tune it.

Fallback chains

A circuit breaker tells you a service is unavailable. It does not tell you what to do instead. That is a design decision, and it should be made before the outage rather than during one.

A fallback chain is an ordered list of increasingly degraded options:

def summarize(document: Document) -> Summary:
    """Summarize with a fallback ladder, degrading rather than failing."""
    try:
        return summarize_with_primary(document)
    except (ProviderError, ProviderOverloadedError):
        logger.warning("primary model unavailable, trying secondary")

    try:
        return summarize_with_secondary(document)
    except (ProviderError, ProviderOverloadedError):
        logger.warning("secondary model unavailable, trying cache")

    cached = cache.get(document.content_hash)
    if cached is not None:
        return Summary.from_cache(cached, is_stale=True)

    return Summary.unavailable(
        document_id=document.id,
        reason="no summarization service available",
    )

Each step is worse than the one before and better than nothing. The final step is the important one: honest degradation, meaning a response that clearly says what could not be done rather than a fabricated one or an unhandled crash.

The is_stale flag on the cached result and the explicit unavailable result both exist so that callers, and eventually users, can tell the difference between a fresh answer and a degraded one. A fallback that silently returns worse data without saying so is worse than a failure, because nobody knows to distrust it.

Designing a degradation ladder in advance

Write the ladder down before you need it, because outages are a bad time to be deciding what your product should do.

For each capability, answer four questions. What is the ideal response? What is an acceptable degraded response? What is the minimum honest response? And what must never happen?

For the recipe extractor:

That last row is the one worth arguing about with your team, and it is the reason to write the table. In an outage, "return something plausible" is a tempting shortcut, and having previously agreed that it is forbidden is what stops it.

Two practical requirements. Every level must be observable, so your logs and metrics record which level served each request, or you will not know you have been running degraded for a week. And every level should be testable, meaning you can force it in a test rather than waiting for a real outage to find out whether the fallback works. A fallback path that has never executed is not a fallback, it is an untested branch.