Lesson 3: Cleaning the corpus before it trains the model

Module 8 · Lesson 3 · About 55 minutes · Needs: Lesson 1 of this module (split, corpus report, BPE, packing) and Module 6 (the proxy and train)

The situation

Sol 421. Lesson 1's corpus report left Kofi Mensah with a question: 18 of the 200 lookup answer lines appear word for word in training documents. "Is the lookup score contaminated, or is that chance?" Meanwhile Tariq Haddad has pulled back the relay archive, Earth's mirror of the station record, as extra text. It arrived full of copies and garbled transmissions.

This lesson cleans the archive and answers Kofi. The running example is lookup question 4, whose answer line "Answer: Ana Ibarra works in the machine shop." also sits in training document 4420.