Incident: A decontamination filter that skipped long documents

Module 8 · Incident file · About 35 minutes

What happened

In May 2020 Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah and 30 colleagues at OpenAI posted "Language Models are Few-Shot Learners", the GPT-3 paper (arxiv.org/abs/2005.14165, version 1 on 28 May 2020, version 4 on 22 July 2020). Before training, the team tried to remove from the training data any text that also appeared in the development and test sets of the benchmarks they would report.

It did not fully work. Section 2.2 says: "Unfortunately, a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model." Appendix C names the failure: "Due to a bug revealed by this analysis, filtering described above failed on long documents such as books." The model had been trained for 300 billion tokens, about 3.14 × 10^23 FLOPs (Table D.1).

The cost fell on the results. The 4 Wikipedia language modelling benchmarks measured in GPT-2, plus the Children's Book Test, turned out "to be almost entirely contained in our training data", so the team did "not report results on these datasets, even though we intended to when starting this work". Penn Tree Bank, unaffected because of its age, "became our chief language modeling benchmark". PIQA and Winograd results carry an asterisk: 132 Winograd schemas "were in fact present in our training set". For most benchmarks the authors found that performance on a clean subset changed "only negligibly".