CourseAI System Design: The Complete Course · Module 3: Capacity and Cost Estimation · part 27 of 107
Part 27 · Module 3: Capacity and Cost Estimation

Incident: A test job ran up 72,000 dollars overnight

28 min read·
Module 3 · Incident file · About 30 minutes

What happened

TIMELINE A $7 budget, a bill near $72,000 March 2020 test deployed 27 March about $5,000 20 minutes $25,000 two hours near $72,000 8 to 10 Dec posts, report later, Google waived the bill as a one-time gesture
A test with a $7 budget became a bill near $72,000 within hours of being noticed (red); Google later waived it, which The Register warned cannot be relied on.

In March 2020 Milkie Way, a small startup, was building a product called Announce. Its founder, Sudeep Chauhan, told the story himself in two posts on 8 December 2020, "We Burnt $72K testing Firebase + Cloud Run and almost went Bankrupt" (parts 1 and 2), and The Register reported it on 10 December 2020 ("Google Cloud (over)Run: How a free trial experiment ended with a $72,000 bill overnight").

The team wanted a web scraper that would not hit Cloud Run's request timeout. The design: each instance scrapes one page and sends every link it finds back to the service as a new request, so pages are scraped in parallel. For the test, Chauhan "created a new GCP project", "set up $7 Cloud Billing budget" and kept the Firebase database "on the Free (Spark) plan". "The worst case we imagined was exceeding the daily free Firestore limits."

A service that feeds itself Milkie Way's scraper on Cloud Run Active flow 1 Processing step number Scraper service Cloud Run Cloud Run instance scrapes one page every link found sent back as requests more instances up to 1,000 1 2 the loop 3 Storage Firestore each write and read
Parallel by design: every page scraped sends its links back as new requests, so the service feeds itself (the loop arrow).

The next day, after a nap, he found emails: the Firebase project had been upgraded to a paid plan, the budget was exceeded, and his card (limited to $100) was declined. The billing page showed about $5,000; "after 5 minutes, the bill read $15,000, in 20 mins, it said $25,000", and "after two hours, it settled at a little short of $72,000". That was Friday 27 March. Google suspended every project on the same card. After the team sent its own incident document, "Google let go of our bill as a one time gesture". The Register's comment: "Such leniency cannot be relied upon."

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan