Incident: The benchmark said quantisation was fine

Module 8 · Incident file · About 30 minutes · Needs: Lesson 1 (INT8 and INT4 post-training quantisation) and Lesson 3 (measuring); the promotion gates in course-guide/03

What happened

On 3 July 2024 researchers at Cohere and Cohere For AI posted "How Does Quantization Affect Multilingual LLMs?" (Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet Üstün, Sara Hooker, Sebastian Ruder; arXiv 2407.03211, version 2 of 12 October 2024, Findings of EMNLP 2024). They quantised their own multilingual models, Command R+ (103 billion parameters), Command R (35 billion) and Aya 23 (35 and 8 billion), and measured what was lost, with automatic benchmarks, an LLM as judge, and human evaluators.

The headline is in the abstract: "automatic metrics severely underestimate" the damage. "A 1.7% average drop in Japanese across automatic tasks corresponds to a 16.0% drop reported by human evaluators on realistic prompts." For French the automatic estimate was -0.3% against -16.6% from human evaluators. Those human figures are for the 103B model at W4-g (4-bit weights with group-wise scaling, using GPTQ) on the authors' internal set of 150 harder prompts.

The damage was uneven. "Non-Latin script languages are more harmed on average": for the 103B model, -0.7% for Latin-script languages against -1.9% for the others. Hard tasks fell fastest: mathematical reasoning dropped 13.1% for the 35B model at W4-g. And sometimes quantisation helped: "an average 1.3% boost across tasks for a 35B model quantized with W8A8".