CourseAgentic AI Engineer: The Complete Course · Module 15: Runtime and Infrastructure · part 87 of 97
Part 87 · Module 15: Runtime and Infrastructure

Incident: Work that timed out faster than it finished

8 min read·13 Oct 2026
Module 15 · Incident file · About 35 minutes

What happened

US-EAST-1, 19 TO 20 OCTOBER 2025 (PDT) 11:48 PM DNS empty 2:25 AM DNS back; leases until 4:14 collapse 4:14 AM throttle, restart 5:28 AM leases back 1:50 PM EC2 normal times from AWS's post-event summary
The DNS fault (red) was fixed by 2:25 AM, but EC2's lease manager then made no progress until engineers throttled its work (blue) at 4:14 AM; recovery (green) followed.

On 19 and 20 October 2025, Amazon Web Services had a long disruption in its N. Virginia (us-east-1) Region. AWS's own post-event summary, "Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region", describes it. The trigger, at 11:48 PM PDT, was "a latent race condition in the DynamoDB DNS management system" that left an empty DNS record for DynamoDB's regional endpoint. DNS was restored by 2:25 AM.

This file is about what happened next, inside EC2. A subsystem called DropletWorkflow Manager (DWFM) keeps a lease on each physical server ("droplet"), checked every few minutes. Those checks depend on DynamoDB, so leases "slowly started to time out". At 2:25 AM, with DynamoDB back, DWFM began to re-establish leases across the EC2 fleet.

DWFM KEEPS A LEASE ON EVERY DROPLET DWFM lease manager droplet droplet droplet DynamoDB down: checks fail no lease: no new instances on that droplet
Lease checks depended on DynamoDB (red), so while it was down every droplet's lease ran out, and each had to be re-established afterwards.

The summary says: "due to the large number of droplets, efforts to establish new droplet leases took long enough that the work could not be completed before they timed out. Additional work was queued to reattempt establishing the droplet lease." DWFM "had entered a state of congestive collapse and was unable to make forward progress". There was "no established operational recovery procedure". At 4:14 AM engineers "throttled incoming work and began selective restarts of DWFM hosts"; by 5:28 AM all leases were back. The summary dates failed new instance launches from 2:25 AM to 10:36 AM, partly from later backlogs elsewhere, and full EC2 recovery to 1:50 PM. AWS writes that the event "impacted many customers in significant ways"; it gives no money figure, and neither does this file.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside