CourseAgentic AI Engineer: The Complete Course · Module 16: Deployment and Operations · part 92 of 97
Part 92 · Module 16: Deployment and Operations

Incident: A blank policy field crashed every region

8 min read·13 Oct 2026
Module 16 · Incident file · About 35 minutes

What happened

GOOGLE CLOUD, 12 JUNE 2025 (PDT) 29 May new check ships ~10:45 blank policy +2 min triage +10 min cause found +40 min red button out ~1 h first status post ~2h 40 us-central-1 times from Google's incident report
The code shipped two weeks earlier (amber); a blank policy (red) triggered it on 12 June, and the red button (green) was fully rolled out within 40 minutes, smaller regions recovering first.

On 12 June 2025 Google Cloud, Google Workspace and Google Security Operations products returned "increased 503 errors in external API requests". Google's incident report, published on its Cloud Service Health page on 13 June 2025, explains why. The incident page lists 55 affected products (fetched 12 October 2026). The report gives no money figure, and neither does this file.

Every Google API request passes through Service Control, which checks authorisation, policy and quota. On 29 May 2025 "a new feature was added to Service Control for additional quota policy checks". The release went out region by region, "but the code path that failed was never exercised during this rollout due to needing a policy change that would trigger the code."

ONE POLICY ROW, EVERY REGION IN SECONDS policy row blank fields region A region B region C crash loop everywhere
The bad policy (red) reached every region within seconds, so every regional Service Control hit the same crash at once.

At about 10:45 PDT on 12 June, a policy change with "unintended blank fields" was inserted, and "this metadata was replicated globally within seconds". Service Control read it, hit a null pointer and went "into a crash loop". Engineers were triaging within 2 minutes and found the cause within 10. The change shipped with a red button to turn off that serving path; it was ready in about 25 minutes and fully rolled out within 40. Large regions took longer: in us-central-1 the restarting tasks "created a herd effect", and full recovery took "up to ~2h 40 mins". The first status post came about an hour in, because the status page was down too.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside