Incident: A whitespace trim took Stack Overflow down

Module 5 · Incident file · About 30 minutes

What happened

On 20 July 2016 Stack Exchange, the company behind Stack Overflow, published "Outage Postmortem - July 20, 2016" on its status blog. It opens: "On July 20, 2016 we experienced a 34 minute outage starting at 14:44 UTC. It took 10 minutes to identify the cause, 14 minutes to write the code to fix it, and 10 minutes to roll out the fix to a point where Stack Overflow became available again."

The cause was text cleaning. "The direct cause was a malformed post that caused one of our regular expressions to consume high CPU on our web servers." The expression was ^[\s‌]+|[\s‌]+$, "intended to trim unicode space from start and end of a line", the same kind of tidying lesson 1 did with .strip(). The post "contained roughly 20,000 consecutive characters of whitespace on a comment line". The post was on the home page list, so the expensive expression ran on every home page view. The home page "is what our load balancer uses for the health check", so "the entire site became unavailable since the load balancer took the servers out of rotation". One post, one regular expression, the whole site down.