CourseLarge Language Models: The Complete Course · Module 10: Evaluation · part 75 of 103
Part 75 · Module 10: Evaluation

Incident: Launch charts that disagreed with their numbers

9 min read·13 Oct 2026
Module 10 · Incident file · About 25 minutes

What happened

GPT-5 LAUNCH, 2025 live slides bars mismatch 7 Aug 'mega chart screwup' on X 7 Aug press reports blog was right 8-11 Aug card: n=477 SWE-bench subset 19 Dec
The chart errors (red) came and went in days; the case count behind the score (amber) was stated months later in the system card.

OpenAI rolled out GPT-5 on Thursday 7 August 2025, with a live presentation. One slide compared scores on SWE-bench Verified, a benchmark of real software issues a model must fix. GIGAZINE, a Japanese technology site, reproduced it on 8 August: GPT-5 scored 52.8% without thinking and 74.9% with it, o3 69.1% and GPT-4o 30.8%. The bar for 52.8 was drawn taller than the bar for 69.1, and the bars for 69.1 and 30.8 were drawn the same height. A second slide, on "coding deception", where lower is better, showed GPT-5's bar labelled 50.0% beside o3's 47.4%, although, by GIGAZINE's account, GPT-5's rate was the lower one. GIGAZINE reported that the chart was nicknamed "vibechart".

SWE-BENCH VERIFIED SLIDE, AS REPORTED model score bar as drawn GPT-5, thinking 74.9% tallest o3 69.1% same as GPT-4o GPT-5, no thinking 52.8% taller than o3 GPT-4o 30.8% same as o3
GIGAZINE's account of the slide: the scores were right, but three bars (red) did not match them.

Julie Bort reported in TechCrunch on 8 August 2025 that "the chart presented a lower benchmark score with a much taller bar", that Sam Altman called it a "mega chart screwup" on X, and that "the charts in the published blog post were correct". Tech.co (Conor Cawley, 11 August 2025) quoted Altman on Reddit: "The numbers here were accurate but we screwed up the bar charts in the livestream overnight", and "People were working late and were very tired, and human error got in the way." No source puts a money cost on it. The cost was trust: the slide existed to compare models, and it compared them wrongly in front of the audience it was made for.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside