Summary

Failures come in seven kinds, and the response differs for each. Syntax and runtime errors mean your code is wrong. Logical errors produce no exception and need tests. Data errors mean rejecting a record with a reason. Network and provider errors may be worth retrying, depending on the specific failure. Model errors are the ones where every other layer reports success and only content validation finds the problem.

Exception handling has four blocks, not two. Keep try narrow so it guards only what can fail, use else to separate success handling from risky code, and use finally for cleanup that must happen regardless. A bare except catches keyboard interrupts and your own bugs, converting informative failures into silent wrong behaviour. Chain exceptions with raise ... from so the original cause survives translation, and define a domain exception hierarchy so callers respond to categories rather than to library details.

The most important reliability decision is whether a failure can succeed on another attempt, and the safe default is that it cannot. Retry with exponential backoff, always with jitter, because synchronised retries from many clients keep a recovering service down. Prefer the provider's Retry-After over your own calculation, and cap it.

Every network call gets a timeout, and there is more than one kind. Connect covers establishing the connection, read covers waiting between response chunks, and total is the backstop that catches a response trickling slowly forever. Timeouts and retries multiply, so give the whole operation a budget rather than only each attempt.

Retries handle a failure, and circuit breakers handle an outage. Closed, open, and half open, with a threshold to trip and a cooldown before testing again. Behind the breaker sits a fallback chain that degrades deliberately: primary, then secondary, then cache marked as stale, then an honest statement that the capability is unavailable. Write that ladder down before the outage, and make each level observable and testable.

Retries create duplicates, so external calls need idempotency keys generated once per logical operation and reused across attempts, ideally derived from content so they survive a restart. At every trust boundary, validate the shape, bound the size and time, and check the semantics you actually depend on, including that the number of results matches the number of inputs.

Key takeaways

• Classify the failure before deciding what to do about it
• A bare except catches Ctrl-C and your own typos
• raise ... from is the difference between a traceable cause and a mystery
• Default to not retrying, so unretryable failures fail fast and visibly
• Backoff without jitter synchronises your clients and prolongs the outage
• The provider's Retry-After beats your calculation, but cap it
• A call without a timeout is not slow in the bad case, it is infinite
• Timeouts and retries multiply, so budget the whole operation
• A circuit breaker converts a slow expensive outage into a fast cheap one
• Decide your degradation ladder before you need it, and forbid fabrication explicitly
• Generate the idempotency key once per operation, not once per attempt
• Check that the number of results matches the number of inputs

Common mistakes to remember

• Wrapping a whole function body in one try block
• Ordering except clauses from general to specific, so the specific ones never run
• Using a bare except, or except Exception without re-raising
• Raising a domain error without from exc and losing the cause
• Retrying a 400, 401, or 403
• Retrying without jitter
• Sleeping for an uncapped Retry-After
• Omitting a timeout because the library seemed to have a default
• Setting a per-attempt timeout with no total budget
• Sharing one circuit breaker across unrelated dependencies
• Returning degraded data without marking it as degraded
• Never testing the fallback path
• Generating an idempotency key inside the retried function
• Trusting a 200 status as evidence that the content is correct
• Accepting a response without checking that the result count matches the request