CourseAI System Design: The Complete Course · Module 8: Orchestration and State · part 59 of 107
Part 59 · Module 8: Orchestration and State

Incident: A retried weekly email went out twice

13 min read·
Module 8 · Incident file · About 30 minutes

What happened

TIMELINE Three hours to a fix LAUNCHNOTES STATUS PAGE, 19 TO 20 JULY 2024 (UTC) 19 Jul 18:44 duplicates reported 19:36 issue identified 21:47 fix deployed 20 Jul 02:00 fix confirmed no count of people or copies, and no cost, was published
About three hours from the first report (red) to a deployed fix (green), and seven to confirmation. The page publishes no numbers for the damage.

On 19 July 2024 LaunchNotes, which makes product roadmap and release-note software, posted an incident on its status page: "Duplicate Notifications for Users (Internal Roadmap weekly roundup email)". The first update, at 18:44 UTC, said management portal users "are receiving duplicate notifications for the Weekly Roadmap roundup email" and that "our system is inadvertently sending out multiple copies of the same notification to affected users".

At 19:36 the team had identified the issue. At 21:47 "the code fix for this issue has been deployed", and at 02:00 UTC on 20 July "Fix has been confirmed". The write-up limits the impact: "Some users that have subscribed to receive roadmap reports received duplicates of the same internal email", only internal management portal users, and "no duplicate emails were sent to page subscribers". It gives no count of people or copies and no cost. It is a small incident, which is why it is useful: the whole mechanism fits in three sentences.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan