Incident: A scale factor from the wrong width

Module 2 · Incident file · About 35 minutes

What happened

On 9 July 2024 Michael Moynihan of Google committed a one-line change to gemma_pytorch, Google's reference PyTorch code for its Gemma models: "Using head_dim instead of override scalar value for the 9b model" (commit 03e6575). The removed line set Gemma 2 9B's query_pre_attn_scalar=224, with the comment # hidden_size / num_attention_heads. After the commit the 9B model scaled its attention scores by its real head width, head_dim=256, instead.

Three days later, on 12 July, Daniel Han opened pull request #8444 on llama.cpp, a widely used engine for running models locally (github.com/ggml-org/llama.cpp/pull/8444): "Gemma 9b should use 256 and not 224 (self.config.hidden_size // self.config.num_attention_heads)." He added that "Google engineers confirmed only the 9B is affected and not the 27B", and that the Hugging Face repository for Gemma 2 9B had already been updated. Georgi Gerganov, llama.cpp's maintainer, replied that the inference code also had to change "so that the correct scaling factor is used". On 13 July a user reported that "regenerating Gemma gguf from HF repos is currently broken because of this". Gerganov continued the work as pull request #8473, merged on 14 July 2024 (github.com/ggml-org/llama.cpp/pull/8473).

None of these sources measures how much output quality the wrong factor cost, and no figure is claimed here. What they do show is how far one number travels: checked in October 2026, published derivatives of Gemma 2 9B on the Hugging Face Hub, such as TheDrummer/Tiger-Gemma-9B-v2 and mlx-community/gemma-2-9b-8bit, still list "query_pre_attn_scalar": 224 in their config.json.