CourseAI System Design: The Complete Course · Module 5: Data Architecture · part 39 of 107
Part 39 · Module 5: Data Architecture

Incident: A migration served incomplete data for six days

30 min read·
Module 5 · Incident file · About 30 minutes

What happened

TIMELINE Six days, two bugs April 2024, UTC 15 Apr 02:00 data incomplete 15 Apr 18:39 identified 16 Apr 17:25 second issue 17 Apr account fixed 18 Apr jira_issue fixed 20 Apr 21:14 resolved Postmortem published 14 May 2024
Six days from incomplete dashboards (red) to resolution (green), with a second, quieter bug found on 16 April after the first was fixed.

On 14 May 2024 Atlassian published a postmortem on its Atlassian Analytics status page: "Missing data for some tables in Atlassian Analytics". Between 15 April 2024 at 02:00 UTC and 20 April at 21:00 UTC, "some Atlassian Analytics customers experienced service degradation", which "led to inconsistencies and incomplete data for the Account, DevOps, and Jira tables in their Atlassian Analytics dashboards". Customers first saw it when querying jira_issue data in the us-east-1 region.

The status updates show the shape of the recovery. At 18:39 UTC on 15 April the team had "identified an issue stemming from a maintenance job". By 16 April the backlog had been processed, but at 17:25 UTC they "identified an additional issue". The account table was fixed on 17 April, jira_issue on 18 April, the DevOps tables last; the incident closed at 21:14 UTC on 20 April. The postmortem gives no count of customers or rows affected, and no cost.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan