CourseAI System Design: The Complete Course · Module 8: Orchestration and State · part 57 of 107
Part 57 · Module 8: Orchestration and State

Lesson 4: Long-running work that survives crashes and retries

30 min read·
Module 8 · Lesson 4 · About 55 minutes · Needs: Module 4 lesson 3 (synchronous or asynchronous), Module 5 lesson 1 (idempotent ingestion, the dead-letter list), Module 1 lesson 2 (retries and idempotency keys). No key and no model calls

The situation

Week 8. Nobody has said what happens when Intake's work is interrupted. Funmi asks: "A deploy restarts the workers. What happens to a sync that was half done?"

The running example is one job, sync-t-marlow-2026-10-05: Marlow County Council's drive sync on 5 October, 79 files. 76 index; three uploads are broken (t-marlow-d077, d078, d079). The job ends by opening one helpdesk ticket for Marlow's IT team with tickets.create. Leila wants one ticket, not two.

One sync, two ways to fail sync-t-marlow-2026-10-05 Active flow Background or async 1 Processing step number API Layer API returns a job id Queue Redis queue intake Worker RQ worker 79 files Ticket system tickets.create one ticket 1 2 3 Ways to fail worker restarts half done? retry two tickets? 76 index, 3 broken uploads: d077, d078, d079
Marlow's sync is one background job (the top row, Northdesk's normal path): 79 files and one ticket at the end, and two ways it can go wrong (pale red): a worker that dies half way and a retry that opens the ticket twice. Numbers give the order.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan