Part 8 · Module 1: Engineering Floor

Lesson 2: The backend behind one late claim

20 min read·13 Oct 2026

Module 1 · Lesson 2 · About 60 minutes · Needs: Lesson 1 (Python you can read and run)

The situation

It is the week before you start at Harbourline. Nadia Okafor forwards one real-looking email from the claims inbox, MSG-3101, and says: "Everything you build has to handle this one."

Dana Price at Northgate writes that HF-88102 "was promised for the 3rd and turned up on the 9th". Something must fetch the shipment, check Northgate's tier in the CRM, record the exception once, triage it without making a clerk wait, and hear when the CRM changes. That something is a backend. This lesson builds a small one and follows Dana's email through it.

A real case, first

On 21 October 2018, maintenance on network equipment cut the link between GitHub's US East Coast hub and its main East Coast data centre. "Connectivity between these locations was restored in 43 seconds." In that window, GitHub's database failover tool, Orchestrator, began directing writes to the West Coast. The clusters then held writes that were not present in the other data centre, so GitHub chose to "prioritize data integrity over site usability and time to recovery" (Jason Warner, GitHub, "October 21 post-incident analysis", 2018).

Service was degraded for 24 hours and 11 minutes. While the databases were restored, "over five million hook events and 80 thousand Pages builds" were queued. About 200,000 webhook payloads "had outlived an internal TTL and were dropped" until engineers paused processing and raised the TTL.

Databases, queues, backlogs and lost webhooks: every part of that story is in this lesson.

A 43-second network break left GitHub degraded for 24 hours 11 minutes, with over five million webhook events queued and about 200,000 dropped (red marks failure, green recovery).

What you will be able to do

Read an HTTP exchange and choose the right status code and auth response.
Build an API with FastAPI that is safe to retry, and call another API that limits you.
Model claims data as related SQL tables and query them safely.
Move slow work to a background job, a cache and a queue, and say what each can lose.
Receive a webhook that is signed, deduplicated and answered fast.

The backend for Dana's claim: an API, SQL tables, a CRM client with a cache, a job queue feeding the model (amber), and a webhook receiver.

Blue boxes are your system; the amber box is the model, the one part whose answer is uncertain. Each section adds a box.

python

%pip install -q -U langchain langgraph langchain-groq langchain-google-genai langchain-ollama

python

import os
from langchain.chat_models import init_chat_model

PROVIDER = "groq"   # "groq" or "gemini" in Colab; "ollama" on your own computer
MODELS = {"groq": "groq:openai/gpt-oss-20b", "gemini": "google_genai:gemini-3.5-flash-lite", "ollama": "ollama:llama3.2:3b"}
KEY = {"groq": "GROQ_API_KEY", "gemini": "GOOGLE_API_KEY", "ollama": None}[PROVIDER]
if KEY and KEY not in os.environ:
    from google.colab import userdata   # keys live in Colab's Secrets panel
    os.environ[KEY] = userdata.get(KEY)
llm = init_chat_model(MODELS[PROVIDER], temperature=0, max_retries=6)

Code walkthrough The course's standard setup cell: PROVIDER picks the host, the key comes from Colab Secrets, and llm is used once, in section 5. max_retries=6 waits out rate limits (429); it does not retry a rejected reply.

python

def ask(model, messages):
    """Returns (reply, None), or (None, reason) after two refusals."""
    for attempt in range(2):                     # one retry: the next reply is usually fine
        try:
            return model.invoke(messages), None
        except Exception as err:
            last = err
    return None, f"provider_error: {type(last).__name__}"

Code walkthrough ask (course-guide/02) retries a reply the provider rejects once, then returns no reply and the reason provider_error.

python

%pip install -q fastapi httpx fakeredis

python

TMS_CUSTOMERS = {
    "C-1042": "NORTHGATE BUILDING SUPPLIES LTD",
    "C-1077": "Pinder & Sons Hardware",
    "C-1130": "GREENWAY GARDEN CTR",
    "C-1188": "Atlas Office Furniture",
    "C-1203": "ST. MARY'S HOSPITAL TRUST",
}

CRM_ACCOUNTS = [
    {"account_id": "ACC-501", "name": "Northgate Building Supplies", "contact": "dana@northgate.example", "tier": "gold"},
    {"account_id": "ACC-502", "name": "Pinder and Sons", "contact": "orders@pinder.example", "tier": "standard"},
    {"account_id": "ACC-503", "name": "Greenway Garden Centre", "contact": "sam@greenway.example", "tier": "standard"},
    {"account_id": "ACC-504", "name": "Atlas Office Furniture Ltd.", "contact": "ops@atlas.example", "tier": "gold"},
    {"account_id": "ACC-505", "name": "St Marys Hospital NHS Trust", "contact": "supplies@stmarys.example", "tier": "gold"},
]

INBOX = [
    {"id": "MSG-3101", "from": "dana@northgate.example", "subject": "HF-88102 late again",
     "body": "Hi, HF-88102 was promised for the 3rd and turned up on the 9th. Call me on 07700 900123 if you need anything. Dana Price, Site Manager"},
]
print(len(TMS_CUSTOMERS), "TMS customers,", len(CRM_ACCOUNTS), "CRM accounts,", len(INBOX), "email")

Code walkthrough Slices of the shared, simulated Harbourline data: TMS customers, CRM accounts and Dana's email. It prints 5 TMS customers, 5 CRM accounts, 1 email.

python

SHIPMENTS = {
    "HF-88101": {"customer": "C-1042", "promised": "2026-03-02", "delivered": "2026-03-02", "freight_charge": 180.00, "declared_value": None, "weight_kg": 420},
    "HF-88102": {"customer": "C-1042", "promised": "2026-03-03", "delivered": "2026-03-09", "freight_charge": 240.00, "declared_value": None, "weight_kg": 610},
    "HF-88145": {"customer": "C-1077", "promised": "2026-03-04", "delivered": "2026-03-04", "freight_charge": 95.00, "declared_value": None, "weight_kg": 60},
    "HF-88190": {"customer": "C-1130", "promised": "2026-03-05", "delivered": None, "freight_charge": 130.00, "declared_value": None, "weight_kg": 150},
    "HF-88207": {"customer": "C-1188", "promised": "2026-03-06", "delivered": "2026-03-06", "freight_charge": 310.00, "declared_value": 6500.00, "weight_kg": 380},
    "HF-88233": {"customer": "C-1203", "promised": "2026-03-09", "delivered": "2026-03-10", "freight_charge": 75.00, "declared_value": None, "weight_kg": 25},
    "HF-88260": {"customer": "C-1077", "promised": "2026-03-10", "delivered": "2026-03-10", "freight_charge": 110.00, "declared_value": None, "weight_kg": 90},
    "HF-88301": {"customer": "C-1188", "promised": "2026-03-11", "delivered": "2026-03-16", "freight_charge": 420.00, "declared_value": None, "weight_kg": 700},
}

EXCEPTIONS_MASTER = [
    {"ref": "HF-88102", "type": "late", "opened": "2026-03-09", "owner": "Joan", "colour": "amber", "notes": "gold acct - Rafael says credit without arguing", "closed": "2026-03-11"},
    {"ref": "HF-88207", "type": "damage", "opened": "2026-03-09", "owner": "Dev", "colour": "red", "notes": "DV 6500 so above cap, needs Tom. waiting on photos", "closed": None},
    {"ref": "HF-88190", "type": "missing", "opened": "2026-03-13", "owner": "Joan", "colour": "red", "notes": "depot says on trailer 14?? chase Tues", "closed": None},
    {"ref": "HF-88233", "type": "late", "opened": "2026-03-10", "owner": "Priti", "colour": "green", "notes": "1 day only, no credit due. hospital - reply same day anyway", "closed": "2026-03-10"},
    {"ref": "HF-88301", "type": "late", "opened": "2026-03-16", "owner": "Dev", "colour": "amber", "notes": "5 days late. 10% = 42.00", "closed": None},
]
print(len(SHIPMENTS), "shipments,", len(EXCEPTIONS_MASTER), "spreadsheet rows")

Code walkthrough The TMS shipments and Joan's spreadsheet rows; an exception is open when closed is None. It prints 8 shipments, 5 spreadsheet rows.

1. HTTP, REST, status codes and auth (Essential)

Every system you will connect at Harbourline talks HTTP. Read one exchange well and you can debug most integrations from a log line.

An HTTP exchange is a method, a path and headers going in, and a status code, headers and a body coming back.

HTTP is a request and a response. The request carries a method (GET reads, POST creates), a path and headers, one of which says who is asking. The response carries a status code, headers and a body, here JSON.

The first digit of a status code says whose problem it is: 2xx done (green), 4xx the caller's mistake, 5xx the server's; 429 means wait (amber).

The status code's first digit says whose problem it is. 2xx: it worked. 4xx: the caller sent something wrong, and resending it will not help (except 429: wait, then retry). 5xx: the server failed; a later retry might work.

401 means "I do not know who you are"; 403 means "I know you, and you may not"; only Joan's clerk key creates the exception.

Authentication asks "who are you?"; authorisation asks "may you do this?". An unknown key gets 401; a known key without permission gets 403. At Harbourline, identity will come from single sign-on (Module 6).

REST names things as nouns in the path and lets the HTTP method say what to do with them; these are the four endpoints this lesson builds.

REST is a convention: things are nouns in the path, and the method says what to do with them. FastAPI is the Python framework used here; its TestClient calls the API in memory, with no server.

python

from fastapi import FastAPI, Header, HTTPException
from fastapi.testclient import TestClient

API_KEYS = {"key-joan-demo": "clerk", "key-board-demo": "viewer"}   # demo values, not secrets
app = FastAPI()

def role_of(authorization):
    token = (authorization or "").removeprefix("Bearer ")
    if token not in API_KEYS:
        raise HTTPException(status_code=401, detail="missing or unknown key")
    return API_KEYS[token]

@app.get("/shipments/{ref}")
def get_shipment(ref: str, authorization: str | None = Header(default=None)):
    role_of(authorization)
    if ref not in SHIPMENTS:
        raise HTTPException(status_code=404, detail=f"no shipment {ref}")
    return {"ref": ref, **SHIPMENTS[ref]}

client = TestClient(app)
JOAN = {"Authorization": "Bearer key-joan-demo"}
for ref, headers in [("HF-88102", JOAN), ("HF-99999", JOAN), ("HF-88102", {})]:
    r = client.get(f"/shipments/{ref}", headers=headers)
    print(r.status_code, r.json())

Code walkthrough

API_KEYS maps demo keys to roles; role_of strips Bearer and raises 401 for an unknown key.
@app.get("/shipments/{ref}") makes an endpoint; {ref} becomes the argument and Header() reads the header. HTTPException becomes an error response; a returned dictionary becomes a 200 with JSON.
The loop prints 200, then 404 for an unknown shipment, then 401 with no key.

2. Building an API that is safe to retry (Essential)

Clients retry, and your own code will retry the CRM in section 3. If a retry creates a second claim, Harbourline may pay twice.

A call can fail before it connects, half way through, or after the work is done but before the reply arrives (amber: the client cannot know which).

Stripe's Brandur Leach lists three ways a call can fail: before it connects, half way through, or after the work succeeds "but the connection break before the server can tell its client about it" (Stripe, 2017). In the last case the client cannot know, so it sends again.

Sent twice without a key, Dana's claim becomes two records (red); with an idempotency key both sends return the same EX-001 (green).

The fix is an idempotency key: a unique value the client sends, and repeats on a retry. The server stores the result under the key and returns it instead of acting twice. For Dana's claim, the message id MSG-3101 is a natural key.

python

from typing import Literal
from pydantic import BaseModel

class NewException(BaseModel):
    ref: str
    type: Literal["damage", "late", "missing", "wrong_address"]
    message_id: str

OPENED, BY_KEY = [], {}

@app.post("/exceptions", status_code=201)
def open_exception(body: NewException, authorization: str | None = Header(default=None),
                   idempotency_key: str | None = Header(default=None)):
    if role_of(authorization) != "clerk":
        raise HTTPException(status_code=403, detail="clerks only")
    if idempotency_key in BY_KEY:
        return BY_KEY[idempotency_key]            # a retry: same answer, nothing new
    record = {"id": f"EX-{len(OPENED) + 1:03d}", **body.model_dump()}
    OPENED.append(record)
    if idempotency_key:
        BY_KEY[idempotency_key] = record
    return record

claim = {"ref": "HF-88102", "type": "late", "message_id": "MSG-3101"}
for attempt in (1, 2):                            # the same request sent twice
    r = client.post("/exceptions", json=claim, headers={**JOAN, "Idempotency-Key": "MSG-3101"})
    print("attempt", attempt, r.status_code, r.json()["id"])
print("viewer:", client.post("/exceptions", json=claim, headers={"Authorization": "Bearer key-board-demo"}).status_code)
print("bad type:", client.post("/exceptions", json={**claim, "type": "lost"}, headers=JOAN).status_code)
print("records stored:", len(OPENED))

Code walkthrough

NewException is a Pydantic model describing the body; Literal[...] allows only the four types.
status_code=201 means "created"; idempotency_key reads the Idempotency-Key header. A seen key returns the stored record.
Both attempts print 201 EX-001; the viewer gets 403; type "lost" gets 422, written by FastAPI. One record is stored.

Break it: retry without the key

python

for attempt in (1, 2):
    r = client.post("/exceptions", json=claim, headers=JOAN)     # no Idempotency-Key
    print("attempt", attempt, r.status_code, r.json()["id"])
print("records stored:", len(OPENED))

Code walkthrough Without the header the server cannot tell a retry from a new claim: EX-002, EX-003, records stored: 3. With a payment attached, that is a wrongly paid claim.

3. Consuming an API that limits you (Essential)

Harbourline's CRM is a REST API with rate limits, and the triage needs it for every claim. You cannot reach it from Colab, so the next cell builds a simulated stand-in allowing three calls a second.

The fourth CRM call inside one second gets 429 with Retry-After: 1; the client waits 1.1 seconds and the lookup succeeds, six calls for five accounts.

Exceed a rate limit and the server answers 429, often with a Retry-After header saying how long to wait. In the run, the fourth lookup inside one second (ACC-504) got 429; the client waited 1.1 seconds and retried.

python

import time, random
crm = FastAPI()
recent = []

@crm.get("/accounts/{account_id}")
def crm_account(account_id: str):
    now = time.monotonic()
    recent[:] = [t for t in recent if now - t < 1.0]
    if len(recent) >= 3:                              # simulated limit: 3 calls per second
        raise HTTPException(429, "rate limit", headers={"Retry-After": "1"})
    recent.append(now)
    account = next((a for a in CRM_ACCOUNTS if a["account_id"] == account_id), None)
    if account is None:
        raise HTTPException(404, "no such account")
    return account

crm_client = TestClient(crm)   # the real CRM: httpx.Client(base_url=..., headers=..., timeout=5.0)

Code walkthrough

recent keeps call times from the last second; a fourth call gets 429 with Retry-After: 1, an unknown id 404.
Against the real CRM, httpx.Client takes the base URL, auth header and a timeout, so a hung call fails after five seconds instead of never.

Retry only what can succeed later: 429, 5xx and timeouts; a 400, 401, 403 or 404 fails the same way every time.

The status code decides whether to retry. A 429, 5xx or timeout may succeed later. A 400, 401, 403 or 404 will fail again; retrying only adds load.

Exponential backoff doubles each wait (0.5, 1, 2, 4 s); a little random jitter, in amber, stops many clients returning in one wave.

Without Retry-After, wait longer after each failure: exponential backoff. Add a random amount, jitter, so clients that failed together do not return together, the "thundering herd" Stripe warns about.

python

crm_calls = 0

def get_account(account_id, tries=4):
    global crm_calls
    for attempt in range(tries):
        crm_calls += 1
        r = crm_client.get(f"/accounts/{account_id}")
        if r.status_code == 200:
            return r.json()
        if r.status_code == 429 or r.status_code >= 500:
            wait = float(r.headers.get("Retry-After", 0.5 * 2 ** attempt)) + random.uniform(0, 0.2)
            print(f"  {account_id}: {r.status_code}, waiting {wait:.1f}s")
            time.sleep(wait)
            continue
        r.raise_for_status()                          # other 4xx: retrying will not help
    raise RuntimeError(f"gave up on {account_id}")

random.seed(16)
t0 = time.monotonic()
tiers = {a: get_account(a)["tier"] for a in ["ACC-501", "ACC-502", "ACC-503", "ACC-504", "ACC-505"]}
print(tiers, f"| {crm_calls} calls in {time.monotonic() - t0:.1f}s")

Code walkthrough

Up to four tries. On 429 or 5xx, wait is Retry-After if given, else 0.5, 1, 2, 4 seconds, plus random.uniform jitter. raise_for_status() raises at once for other errors.
The run waits once, for ACC-504, and prints five tiers, 6 calls, about 1.1 s. Northgate is gold.

4. SQL and relational modelling (Essential)

A spreadsheet that repeats a customer's name in every row needs many edits when it changes, and one missed edit is a wrong answer. A relational database stores each fact once.

Three tables, each fact stored once and linked by foreign keys: an exception points at a shipment, a shipment at a customer.

Relational modelling means one table per kind of thing. A primary key names each row. A foreign key points at another table's key, so an exception holds HF-88102, not a copy of the shipment. SQLite ships with Python; services use PostgreSQL.

python

import sqlite3
db = sqlite3.connect(":memory:")
db.executescript("""
PRAGMA foreign_keys = ON;
CREATE TABLE customers  (code TEXT PRIMARY KEY, tms_name TEXT NOT NULL);
CREATE TABLE shipments  (ref TEXT PRIMARY KEY, customer TEXT NOT NULL REFERENCES customers(code),
                         promised TEXT, delivered TEXT, freight_charge REAL);
CREATE TABLE exceptions (id INTEGER PRIMARY KEY, ref TEXT NOT NULL REFERENCES shipments(ref),
                         type TEXT, owner TEXT, opened TEXT, closed TEXT);
""")
db.executemany("INSERT INTO customers VALUES (?, ?)", TMS_CUSTOMERS.items())
db.executemany("INSERT INTO shipments VALUES (?, ?, ?, ?, ?)",
               [(ref, s["customer"], s["promised"], s["delivered"], s["freight_charge"]) for ref, s in SHIPMENTS.items()])
db.executemany("INSERT INTO exceptions (ref, type, owner, opened, closed) VALUES (?, ?, ?, ?, ?)",
               [(x["ref"], x["type"], x["owner"], x["opened"], x["closed"]) for x in EXCEPTIONS_MASTER])
for table in ("customers", "shipments", "exceptions"):
    print(table, db.execute(f"SELECT COUNT(*) FROM {table}").fetchone()[0])

Code walkthrough

":memory:" is a throwaway database. PRAGMA foreign_keys = ON makes SQLite enforce REFERENCES; it is off by default.
executemany inserts many rows, filling each ? from a tuple. Counts: 5, 8, 5.

A join walks from each open exception to its shipment and then its customer; for Dev it returns HF-88207 and HF-88301, both Atlas.

A join follows the keys. "What is Dev holding?" starts at his open exceptions, steps to each shipment, then to its customer.

python

OPEN_FOR = """
SELECT e.ref, e.type, c.tms_name, s.freight_charge
FROM exceptions e
JOIN shipments s ON s.ref = e.ref
JOIN customers c ON c.code = s.customer
WHERE e.closed IS NULL AND e.owner = ?
ORDER BY e.opened"""
for row in db.execute(OPEN_FOR, ("Dev",)):
    print(row)
try:
    db.execute("INSERT INTO exceptions (ref, type) VALUES ('HF-00000', 'late')")
except sqlite3.IntegrityError as err:
    print("refused:", err)

Code walkthrough

JOIN ... ON matches rows with equal keys; e, s, c are table aliases. ? takes the owner as a parameter.
Two rows print, both Atlas. An exception for a missing shipment is refused: FOREIGN KEY constraint failed.

Break it: glue the input into the SQL

Glued into the SQL text, "Dev' OR '1'='1" rewrites the query and returns all 5 rows; passed as a parameter it is just an odd name and returns 0.

python

owner = "Dev' OR '1'='1"
glued = OPEN_FOR.replace("e.owner = ?", f"e.owner = '{owner}'")
print("glued into the text:", len(db.execute(glued).fetchall()), "rows")
print("passed as a parameter:", len(db.execute(OPEN_FOR, (owner,)).fetchall()), "rows")

Code walkthrough Glued into the text, the quote rewrites the condition to "... OR true": 5 rows, closed ones too. As a parameter it is just a strange name: 0 rows. This is SQL injection.

5. Async and background jobs (Essential)

Triage calls a model, which can take seconds. A clerk should not stare at a spinner, and an API holding a connection per slow call runs out of them.

Five 300 ms lookups take 1.5 s one after another and 0.3 s when they wait together; async removes the queueing, not the wait.

Most backend time is waiting on the network. Async code lets one program wait on many calls at once. It does not make a call faster; it stops calls queueing behind each other.

python

import asyncio
from concurrent.futures import ThreadPoolExecutor

async def crm_lookup(account_id):
    await asyncio.sleep(0.3)                   # stands in for a 300 ms network call
    return account_id

async def one_at_a_time(ids):
    return [await crm_lookup(i) for i in ids]

async def all_at_once(ids):
    return await asyncio.gather(*(crm_lookup(i) for i in ids))

def run(coro):                                 # Colab already runs an event loop, so use a thread there
    try:
        asyncio.get_running_loop()
    except RuntimeError:
        return asyncio.run(coro)
    with ThreadPoolExecutor(1) as pool:
        return pool.submit(asyncio.run, coro).result()

ids = [a["account_id"] for a in CRM_ACCOUNTS]
for fn in (one_at_a_time, all_at_once):
    t0 = time.monotonic()
    run(fn(ids))
    print(f"{fn.__name__}: {time.monotonic() - t0:.1f}s for {len(ids)} lookups")

Code walkthrough

async def makes a function that can pause; await asyncio.sleep pauses without blocking. asyncio.gather starts all five and waits together.
run exists because Colab already runs an event loop, so the work runs on a fresh one in a thread. Output: 1.5s, then 0.3s.

The API answers 202 with a job id at once; a worker takes the job, calls the model (amber) and records the result for the client to read.

When the caller should not wait at all, use a background job: the API records the work, answers 202 Accepted with a job id, and a worker does it later. Redis holds the queue; fakeredis stands in for it.

python

import fakeredis
red = fakeredis.FakeRedis(decode_responses=True)

@app.post("/messages/{message_id}/triage", status_code=202)
def request_triage(message_id: str, authorization: str | None = Header(default=None)):
    role_of(authorization)
    job_id = f"job-{red.incr('job_seq')}"
    red.hset(job_id, mapping={"status": "queued", "message_id": message_id})
    red.lpush("triage_jobs", job_id)
    return {"job_id": job_id, "status": "queued"}

@app.get("/jobs/{job_id}")
def job_status(job_id: str):
    job = red.hgetall(job_id)
    if not job:
        raise HTTPException(404, "no such job")
    return job

t0 = time.monotonic()
job = client.post("/messages/MSG-3101/triage", headers=JOAN).json()
print(job, f"answered in {time.monotonic() - t0:.2f}s")
print(client.get(f"/jobs/{job['job_id']}").json())

Code walkthrough incr numbers the jobs; hset stores the job record; lpush adds its id to the triage_jobs list. The API answers job-1, queued in 0.00s; no worker has run yet.

python

class Triage(BaseModel):
    type: Literal["damage", "late", "missing", "wrong_address"]

triager = llm.with_structured_output(Triage, method="json_schema")

def work_one():
    job_id = red.rpop("triage_jobs")
    if job_id is None:
        return None
    red.hset(job_id, "status", "running")
    message = next(m for m in INBOX if m["id"] == red.hget(job_id, "message_id"))
    result, why = ask(triager, "Classify this freight claim email.\n\n" + message["body"])
    if result is None:                                # refused twice: a clerk types this one
        red.hset(job_id, mapping={"status": "failed", "error": why})
        return job_id
    red.hset(job_id, mapping={"status": "done", "type": result.type})
    return job_id

done = work_one()
print(client.get(f"/jobs/{done}").json())

Code walkthrough with_structured_output(..., method="json_schema") makes the model return one of the four types. work_one takes the oldest job with rpop, calls the model and stores the type with status done; when tested, both Groq and Gemini returned late. If the provider refuses twice, the job is stored as failed with the reason. A real worker loops in its own process.

6. Caching (Recommended)

Every triage needs the customer's tier, and the CRM allows three calls a second. Asking again for a minute-old answer wastes that limit.

With a five-minute TTL, five lookups of Northgate cost two CRM calls: blue misses go to the CRM, green hits are served from the cache.

A cache keeps recent answers close by. Each entry has a TTL (time to live); after it, the next lookup goes to the CRM again.

python

CACHE, TTL = {}, 300                                   # keep CRM answers for 5 minutes

def cached_account(account_id, now):
    entry = CACHE.get(account_id)
    if entry and now - entry["at"] < TTL:
        return entry["account"], "hit"
    account = get_account(account_id)
    CACHE[account_id] = {"at": now, "account": account}
    return account, "miss"

time.sleep(1.0)                                        # let the CRM's one-second window clear
crm_calls = 0
for now in (0, 30, 60, 90, 400):                       # seconds into the morning (a simulated clock)
    account, how = cached_account("ACC-501", now)
    print(f"t={now:>3}s {how:4}  tier={account['tier']}")
print("CRM calls:", crm_calls, "for 5 lookups")

Code walkthrough A hit is an entry younger than TTL; a miss calls get_account and stores the answer with its time. now is a simulated clock, so every run agrees: miss, hit, hit, hit, miss, CRM calls: 2.

When the CRM changes, the cache keeps answering "gold" (red, wrong) until the entry expires at 700 s; the TTL sets how long you can be wrong.

Break it: serve a stale tier

python

CRM_ACCOUNTS[0]["tier"] = "standard"                   # at t=410 the CRM moves Northgate off gold
for now in (420, 600, 720):
    account, how = cached_account("ACC-501", now)
    print(f"t={now:>3}s {how:4}  tier={account['tier']}")
CRM_ACCOUNTS[0]["tier"] = "gold"                       # put the shared data back

Code walkthrough At 420 s and 600 s the cache still answers gold; only at 720 s does standard appear. Choose a TTL by asking what a wrong answer costs. Section 8 fixes this with an event.

7. Message queues (Recommended)

The triage list in section 5 is already a queue. Knowing that matters, because a queue changes what can be lost.

A queue sits between producers and consumers, so either side can be slow, busy or down; a second worker (grey, optional) drains a backlog faster.

A message queue sits between producers, which add messages, and consumers, which take them. The API accepts work while the worker is busy or down. GitHub's five million queued events were a queue doing its job.

A worker that crashes after a plain pop loses the job (red); one that first moves it to a processing list can have it put back (green), so it may run twice.

The danger is between taking a message and finishing it. A worker that pops job-a and crashes loses it. Moving it to a "processing" list first lets a sweeper put it back, so it may run twice: at-least-once delivery. Consumers must be idempotent.

python

red.lpush("q", "job-a")
taken = red.rpop("q")                                  # worker takes job-a ... and crashes here
print("plain pop, after crash:", red.lrange("q", 0, -1), red.lrange("processing", 0, -1))

red.lpush("q", "job-a")
taken = red.lmove("q", "processing", "RIGHT", "LEFT")  # take it, but park a copy
print("moved, after crash:", red.lrange("q", 0, -1), red.lrange("processing", 0, -1))
red.lmove("processing", "q", "LEFT", "RIGHT")          # a sweeper puts it back
print("after the sweeper:", red.lrange("q", 0, -1))

Code walkthrough After rpop and a crash both lists are empty: the job is gone. lmove moves it to processing in one step, so it survives: [] ['job-a'], and the sweeper puts it back. A finished worker removes it with lrem.

Production queues such as RQ on Redis do this for you. On your own computer (not run here, verified against the RQ workers documentation, checked October 2026):

bash

docker run -d --name redis -p 6379:6379 redis:7
pip install rq
rq worker --url redis://localhost:6379 triage

Code walkthrough The first line runs Redis in a container; the last starts a worker on the triage queue. --burst would make it stop when the queue is empty.

8. Webhooks and event handling (Recommended)

Section 6 left Northgate's tier stale for minutes. The CRM can instead tell you the moment it changes.

Polling asks "anything new?" on a timer and mostly hears no; a webhook is told the moment something changes, but your endpoint is now open to callers.

Polling asks "anything new?" on a timer. A webhook reverses it: the other system POSTs to your URL when an event happens. Faster and cheaper, but strangers can now reach your endpoint.

A receiver checks the signature, skips deliveries it has seen, queues the event and answers within seconds; the worker does the rest.

GitHub's documentation gives a receiver's rules: answer 2xx "within 10 seconds" and queue the work; check an HMAC-SHA256 signature (X-Hub-Signature-256) in constant time; use the delivery id (X-GitHub-Delivery) to spot duplicates. The CRM's header names below are made up.

python

import hmac, hashlib, json
from fastapi import Request

WEBHOOK_SECRET = b"demo-secret-not-real"   # a real one lives in a secret store (Lesson 4)
seen = set()

@app.post("/webhooks/crm", status_code=202)
async def crm_webhook(request: Request):
    body = await request.body()
    expected = "sha256=" + hmac.new(WEBHOOK_SECRET, body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, request.headers.get("X-Signature-256", "")):
        raise HTTPException(401, "bad signature")
    delivery = request.headers.get("X-Delivery-Id")
    if delivery in seen:
        return {"status": "duplicate, ignored"}
    seen.add(delivery)
    event = json.loads(body)
    CACHE.pop(event["account_id"], None)       # the cached copy is now out of date
    red.lpush("crm_events", body)              # store it, answer fast, work later
    return {"status": "queued"}

Code walkthrough

The HMAC is a fingerprint of the raw body that only a secret holder can make; compare_digest compares in constant time.
seen remembers delivery ids. A fresh event drops Northgate from the cache and is queued for the worker.

Three deliveries: a fresh signed event is queued, the sender's retry is ignored as a duplicate, and a forged body is refused with 401.

python

def sign(body):
    return "sha256=" + hmac.new(WEBHOOK_SECRET, body, hashlib.sha256).hexdigest()

body = json.dumps({"event": "account.updated", "account_id": "ACC-501", "tier": "standard"}).encode()
good = {"X-Signature-256": sign(body), "X-Delivery-Id": "d-7001"}
print(client.post("/webhooks/crm", content=body, headers=good).json())
print(client.post("/webhooks/crm", content=body, headers=good).json())      # the sender retried
forged = body.replace(b"standard", b"platinum")
print(client.post("/webhooks/crm", content=forged, headers={**good, "X-Delivery-Id": "d-7002"}).status_code)
print("queued events:", red.llen("crm_events"), "| ACC-501 cached:", "ACC-501" in CACHE)

Code walkthrough sign plays the CRM. The first delivery is queued, the retry duplicate, ignored, the forged body 401. One event is queued and Northgate is no longer cached.

In production

GitHub's analysis shows these parts under strain. The queue held the work while the databases were down. But it had a TTL, and about 200,000 webhook payloads outlived it and were dropped until engineers raised it. GitHub's next steps included testing its failure assumptions. For the engagement, three habits follow: decide what happens to a message that waits too long, and alert rather than drop it silently; make every consumer idempotent, since retries and redeliveries will happen (a redelivered GitHub webhook keeps its X-GitHub-Delivery value for exactly this reason); and test the backlog, not only the happy path.

What you added to the engagement

An API over shipments and exceptions, with 401 and 403 for unknown and unpermitted callers.
Exception creation that is safe to retry, keyed on the message id.
A CRM client that honours timeouts, 429 and Retry-After, with backoff and jitter.
A relational schema with foreign keys and parameterised queries.
Background triage with a job id, a cache with a TTL, and a signed, deduplicated webhook receiver that clears it.

Check yourself

A call returns 403. Should your client retry it?
Why does Idempotency-Key: MSG-3101 stop Dana's claim being recorded twice?
What did the foreign key on exceptions.ref prevent?
When does an endpoint answer 202 instead of 201?
Why must a webhook consumer be idempotent even with a seen set?

No. It will fail the same way every time. Retry 429, 5xx and timeouts.
The server returns the record stored under the key instead of creating another.
An exception for a shipment that does not exist (HF-00000).
When work is accepted but not done yet; 201 means the record exists now.
The seen set is lost on restart, and queues deliver at least once.

Exercise

Add a GET /exceptions/{exception_id} endpoint that returns a record from OPENED, 404 if it does not exist, and 401 without a key. Then run this check:

python

@app.get("/exceptions/{exception_id}")
def get_exception(exception_id: str, authorization: str | None = Header(default=None)):
    role_of(authorization)
    record = next((x for x in OPENED if x["id"] == exception_id), None)
    if record is None:
        raise HTTPException(404, "no such exception")
    return record

assert client.get("/exceptions/EX-001", headers=JOAN).json()["message_id"] == "MSG-3101"
assert client.get("/exceptions/EX-999", headers=JOAN).status_code == 404
assert client.get("/exceptions/EX-001").status_code == 401
print("all checks passed")

Code walkthrough One working answer; write your own first and keep the assert lines, which test the found, missing and unauthenticated cases.

Key terms

Term	Meaning
Status code	Three digits on every response: 2xx done, 4xx caller's error, 5xx server's error
Authentication / authorisation	Who you are (401 if unknown) / what you may do (403 if not allowed)
REST	Things as nouns in the path, actions as HTTP methods
Idempotency key	A client-chosen id that makes a retried request act only once
Rate limit	A cap on calls per period; exceeding it returns 429
Foreign key	A column that must match a row in another table
SQL injection	Input glued into a query that changes its meaning
Background job	Work recorded by the API and done later by a worker
TTL	How long a cached entry or queued message is kept
Webhook	An HTTP POST another system sends you when something happens

Sources

Jason Warner, GitHub, "October 21 post-incident analysis", 2018 (updated 2021). https://github.blog/news-insights/company-news/oct21-post-incident-analysis/
Brandur Leach, Stripe, "Designing robust and predictable APIs with idempotency", 2017. https://stripe.com/blog/idempotency
GitHub Docs, "Best practices for using webhooks", checked October 2026. https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks
GitHub Docs, "Validating webhook deliveries", checked October 2026. https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries
RQ, "Workers", checked October 2026. https://python-rq.org/docs/workers/