Lesson 4: Background Work

When a request must return before the work finishes

Some work outlives a reasonable request. Extracting from a 200-page document, re-embedding a corpus after a model change, or generating a report over ten thousand records all take minutes to hours.

Three signs the work belongs in the background: it takes longer than a client will wait, which is roughly thirty seconds; it should survive the client disconnecting; or it should be retried independently of the request that started it.

FastAPI's built-in background tasks are not the answer for these. They run in the same process after the response is sent, so they die with a deployment, cannot be retried, cannot be monitored, and have no status. They are fine for a fire-and-forget side effect such as sending a notification, and wrong for work whose completion matters.

Task queues, status endpoints, and callbacks

The pattern has three parts: submit and return an identifier, expose status, and optionally notify on completion.

@router.post("/v1/jobs/extract-corpus", status_code=status.HTTP_202_ACCEPTED)
async def submit_corpus_extraction(
    body: CorpusExtractionRequest,
    deps: Dependencies = Depends(get_deps),
    user: AuthenticatedUser = Depends(require_user),
) -> JobAccepted:
    """Queue a long-running extraction and return immediately."""
    job = Job(
        id=str(uuid.uuid4()),
        kind="extract_corpus",
        payload=body.model_dump(),
        user_id=user.id,
        status="queued",
        idempotency_key=body.idempotency_key,
    )
    existing = await deps.jobs.find_by_idempotency_key(job.idempotency_key)
    if existing is not None:
        return JobAccepted(job_id=existing.id, status=existing.status)

    await deps.jobs.enqueue(job)
    return JobAccepted(job_id=job.id, status="queued")

202 Accepted, not 200, because the work has not happened yet and the status code should say so.

The idempotency key is Module 5's mechanism applied to job submission. A client retrying a submission that succeeded but whose response was lost must not start a second job.

Status endpoint:

@router.get("/v1/jobs/{job_id}")
async def get_job(job_id: str, deps=Depends(get_deps), user=Depends(require_user)) -> JobStatus:
    job = await deps.jobs.get(job_id)
    if job is None or job.user_id != user.id:
        raise HTTPException(status_code=404, detail="job not found")
    return JobStatus(
        job_id=job.id,
        status=job.status,                    # queued, running, succeeded, failed
        progress=job.progress,                # 0.0 to 1.0 where known
        result_url=job.result_url,
        error=job.error,
        cost_usd=str(job.cost_usd) if job.cost_usd else None,
    )

Note the ownership check combined into the 404 rather than a 403, so a user cannot learn that someone else's job exists.

Webhook callbacks notify a client rather than making them poll.

async def notify_completion(job: Job, deps: Dependencies) -> None:
    """Call the client's webhook, signed and with retries."""
    if not job.callback_url:
        return
    payload = {"job_id": job.id, "status": job.status, "completed_at": _now_iso()}
    body = json.dumps(payload)
    signature = hmac.new(deps.webhook_secret, body.encode(), hashlib.sha256).hexdigest()

    await post_with_retry(
        job.callback_url,
        content=body,
        headers={"content-type": "application/json", "x-signature": signature},
    )

Three requirements. Sign the payload so the receiver can verify it came from you. Retry with backoff, since the receiver may be briefly down, using Module 5's decorator. And keep the payload minimal, carrying an identifier rather than the result, so the client fetches it over an authenticated channel rather than receiving data over a URL they gave you.

The worker. Whether you use a dedicated queue system or a database-backed queue, the requirements are the same and they are all things earlier modules built: idempotent processing so a redelivered message is harmless, checkpointing from Module 4 so a long job resumes, bounded concurrency from Module 8 so workers do not exhaust the provider, cost attribution from Module 10 so job spend is visible, and graceful shutdown so a deployment does not lose in-progress work.

[IMAGE PROMPT M12-3
Purpose: Show the submit, poll, and callback pattern for background work and where each guarantee comes from.
Visual type: Sequence diagram with three participants.
Prompt: A clean educational sequence diagram with three vertical participant lanes labelled left to right "client", "API", and "worker", plus a fourth narrow lane on the right labelled "job store". Arrows between lanes in time order from top to bottom: from client to API labelled "POST /jobs with idempotency key", from API to job store labelled "enqueue", from API back to client labelled "202 Accepted, job_id", then from worker to job store labelled "claim job, mark running", a self-loop on the worker labelled "checkpointed progress, bounded concurrency", from client to API labelled "GET /jobs/{id}" with a return arrow labelled "status: running, progress 0.4", then from worker to job store labelled "mark succeeded, record cost", and finally from worker to client labelled "signed webhook: job_id and status only". A note beside the 202 arrow reads "returns before the work finishes". A note beside the webhook arrow reads "identifier only, client fetches the result authenticated".
Required elements: Four labelled lanes, time-ordered arrows with the stated labels, a worker self-loop, both the polling exchange and the webhook, two explanatory notes.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, standard sequence diagram conventions.
Layout: Vertical time flow with four lanes reading left to right.
Text labels: "client", "API", "worker", "job store", "POST /jobs with idempotency key", "enqueue", "202 Accepted, job_id", "claim job, mark running", "checkpointed progress, bounded concurrency", "GET /jobs/{id}", "status: running, progress 0.4", "mark succeeded, record cost", "signed webhook: job_id and status only", "returns before the work finishes", "identifier only, client fetches the result authenticated".
Aspect ratio: 4:3
Accessibility: Label every arrow in text and rely on vertical ordering rather than colour to convey sequence.
Avoid: Vendor queue logos, decorative icons, tiny text, watermarks.
Alt text: Sequence diagram showing a client submitting a job with an idempotency key, receiving 202 Accepted with a job id, a worker claiming and checkpointing the job, the client polling for status, and a signed webhook carrying only the job identifier on completion.
END IMAGE PROMPT]