Lesson 3: Chunk size, overlap and the metadata beside them

Module 3 · Lesson 3 · About 55 minutes · Needs: Lessons 1 and 2 of this module (splitters, the evidence check, structure-aware chunking)

The situation

Lesson 2 gave Grace a chunker that follows headings. Before Kwame indexes with it, three people have questions. Kwame asks what size chunks should be, and whether they should overlap. Farah asks how a chunk knows who may read it. Hannah asks what happens when bulletin SB-114 is withdrawn on 8 October: will its chunks leave the index?

This lesson answers with measurements on 15 Hollin documents and the 17 labelled questions they answer. The thread to follow is q30, "how much is the inspection fee for an out-of-warranty repair". All Hollin data is simulated.