Lesson 1: Chunking

The problem this lesson solves

The recipe-extractor corpus now holds two million cleaned documents. A user asks "what temperature should I bake the focaccia at, and for how long?" The system retrieves five passages, sends them to the model, and gets back a confident answer of 180 degrees for 45 minutes.

The recipe says 220 degrees for 25 minutes.

The prompt was not the problem. The model was not the problem. The passage containing the baking instructions was never retrieved, because the document was split at a fixed 500 characters and the split landed between "Bake at 220C" and "for 25 minutes", leaving two fragments that each answered half a question and neither of which matched the query well.

Chunks are the unit of retrieval. You do not retrieve documents, you retrieve pieces of them, and how you cut those pieces determines what can be found. This is the least glamorous decision in a retrieval system and one of the most consequential.

Fixed-size, sentence, paragraph, recursive, and structure-aware

Fixed size splits every N characters or tokens.

def chunk_fixed(text: str, *, size: int = 500) -> list[str]:
    return [text[i : i + size] for i in range(0, len(text), size)]

Predictable and simple, and it cuts mid-word, mid-sentence, and mid-table. Use it as a baseline to beat, not as a solution.

Note that character size and token size are different. A 500 character chunk holds roughly 125 tokens of English and far fewer of a language whose script tokenizes less efficiently. When your budget is in tokens, chunk in tokens.

Sentence splitting cuts at sentence boundaries.

def chunk_sentences(text: str, *, max_chars: int = 500) -> list[str]:
    """Group whole sentences into chunks up to a size limit."""
    chunks: list[str] = []
    current: list[str] = []
    length = 0

    for sentence in split_sentences(text):
        if length + len(sentence) > max_chars and current:
            chunks.append(" ".join(current))
            current, length = [], 0
        current.append(sentence)
        length += len(sentence)

    if current:
        chunks.append(" ".join(current))
    return chunks

Better than fixed size because a chunk is always readable. Sentence detection is harder than splitting on full stops, since abbreviations, decimals, and ellipses all contain them, so use a library rather than a regular expression.

Paragraph splitting cuts on blank lines. Paragraphs are usually semantically coherent, which is the property you want, and they vary enormously in length, so you still need to merge short ones and split long ones.

Recursive splitting is the common default. It tries a hierarchy of separators, falling back only when a chunk is still too large.

DEFAULT_SEPARATORS = ["\n\n", "\n", ". ", " ", ""]


def chunk_recursive(text: str, *, size: int, separators: list[str]) -> list[str]:
    """Split on the largest separator that produces small enough pieces."""
    if len(text) <= size:
        return [text]

    separator, *rest = separators
    if separator == "":
        return [text[i : i + size] for i in range(0, len(text), size)]

    pieces = text.split(separator)
    chunks: list[str] = []
    for piece in pieces:
        if len(piece) <= size:
            chunks.append(piece)
        else:
            chunks.extend(chunk_recursive(piece, size=size, separators=rest))
    return merge_small(chunks, size=size)

The idea is to prefer the largest natural boundary that fits, so paragraphs stay whole where possible and only oversized paragraphs get split at sentences.

Structure-aware splitting uses the document's actual format. A Markdown document has headings, a recipe page has an ingredients section and a method section, and an HTML page has semantic elements. Splitting on those boundaries produces chunks that correspond to something a human would recognise as a unit.

def chunk_markdown(text: str, *, max_chars: int = 800) -> list[Chunk]:
    """Split on headings, keeping the heading path with each chunk."""
    chunks: list[Chunk] = []
    for section in split_on_headings(text):
        body = section.body
        if len(body) <= max_chars:
            chunks.append(Chunk(text=body, heading_path=section.path))
        else:
            for piece in chunk_recursive(body, size=max_chars, separators=DEFAULT_SEPARATORS):
                chunks.append(Chunk(text=piece, heading_path=section.path))
    return chunks

This is the strategy that most often produces a real improvement, and it is the one people skip because it requires knowing the format. For a corpus with consistent structure, which most scraped corpora have within a source, it is worth writing.

Overlap and what it buys

Overlap repeats some content at the boundary between adjacent chunks.

def chunk_with_overlap(text: str, *, size: int = 500, overlap: int = 100) -> list[str]:
    step = size - overlap
    return [text[i : i + size] for i in range(0, len(text), step)]

What it buys is a second chance at boundary content. If "Bake at 220C for 25 minutes" straddles a boundary, an overlap of 100 characters means one chunk probably contains the whole sentence.

What it costs. Storage and embedding cost rise by roughly the overlap fraction, so 20 percent overlap means 20 percent more chunks to embed and store. Retrieval returns near-duplicates, since two overlapping chunks both match the same query, which wastes slots in your top-k and is why the deduplication in Lesson 5 exists. And it is a patch rather than a fix: it reduces the damage from bad boundaries instead of producing good ones.

A reasonable default is 10 to 20 percent, and the honest position is that overlap matters less when your boundaries are good. If structure-aware chunking is producing coherent sections, heavy overlap is compensating for a problem you no longer have.

Chunk boundaries as a silent cause of wrong answers

The failure at the start of this lesson has a specific shape worth recognising, because it does not look like a chunking problem from the outside.

Nothing errors. Retrieval returns five chunks with plausible similarity scores. The model produces a fluent, confident answer. The only symptom is that the answer is wrong, and the natural response is to blame the prompt or the model and start tuning them, which cannot help because the information was never in the context.

Three boundary failures cause most of it.

Split facts. A single statement divided across two chunks, so neither chunk contains the complete claim, and both match the query weakly.

Lost referents. A chunk beginning "Preheat it to 220C" has lost what "it" refers to. The chunk is readable and useless, and it may still be retrieved and still produce a wrong answer.

Orphaned context. A chunk containing a table of baking times with no indication of what is being baked, because the heading was three chunks earlier.

The mitigation for all three is to carry context into the chunk, which the metadata section below covers, and to prefer boundaries the document itself provides.

[IMAGE PROMPT M9-1
Purpose: Show how naive fixed-size splitting destroys document structure and produces chunks that cannot answer a question, compared with structure-aware splitting.
Visual type: Two-panel document splitting comparison.
Prompt: A clean educational comparison with two panels side by side, each showing the same stylised document made of labelled blocks stacked vertically: a heading block reading "Focaccia", a paragraph block, a sub-heading block reading "Baking", a short paragraph block containing the visible text "Bake at 220C for 25 minutes", and a small table block. The left panel is headed "Fixed 500 characters" and shows three horizontal cut lines drawn straight across the document at arbitrary positions, one cutting through the middle of the "Bake at 220C for 25 minutes" line and one cutting through the table. Beside it, three resulting chunk boxes are shown, one containing "...Bake at 220C", another containing "for 25 minutes...", and a third containing half a table with no header row, each annotated "answers nothing on its own". The right panel is headed "Structure aware" and shows cut lines falling only at the heading and sub-heading boundaries, with resulting chunk boxes each carrying a small tag reading "heading: Focaccia > Baking" and one chunk containing the complete sentence and the complete table, annotated "complete unit, retrievable".
Required elements: Identical source document in both panels, arbitrary cuts through content on the left, boundary-aligned cuts on the right, resulting chunk boxes with their annotations, heading path tags on the right chunks.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, document blocks abstract except the one visible sentence.
Layout: Two equal panels side by side with a thin divider, each showing the document on its left and resulting chunks on its right.
Text labels: "Fixed 500 characters", "Structure aware", "Focaccia", "Baking", "Bake at 220C for 25 minutes", "...Bake at 220C", "for 25 minutes...", "answers nothing on its own", "heading: Focaccia > Baking", "complete unit, retrievable".
Aspect ratio: 16:9
Accessibility: Show the difference through cut position, chunk contents, and text annotations rather than colour alone.
Avoid: Real recipe photography, decorative elements, tiny text, logos, watermarks.
Alt text: Comparison showing fixed-size chunking cutting through a sentence and a table producing fragments that answer nothing, against structure-aware chunking cutting at headings and producing complete chunks tagged with their heading path.
END IMAGE PROMPT]

Headings, tables, and code blocks

Three content types that naive splitting damages badly.

Headings carry the context for everything beneath them, and they are usually the shortest line in the document. Split naively, the heading ends up alone in one chunk while its content sits in others with no indication of what they describe.

The fix is to prepend the heading path to every chunk from that section:

def with_heading_context(chunk_text: str, heading_path: list[str]) -> str:
    """Prefix the chunk with its position in the document hierarchy."""
    if not heading_path:
        return chunk_text
    return " > ".join(heading_path) + "\n\n" + chunk_text

A chunk now begins "Focaccia Recipes > Classic Focaccia > Baking" before its content. It costs a few tokens per chunk and makes an enormous difference to both retrieval and to whether the model can interpret the passage.

Tables are the worst case. A table split across chunks loses its header row, so a fragment reads as a grid of numbers with no column names. Keep tables whole even when they exceed your size limit, or repeat the header row on each piece if a table genuinely must be split. Adding a one-line description of what the table contains, generated at index time, is often more retrievable than the table itself.

Code blocks should not be split mid-block. A fragment of code is rarely useful, and a fragment containing an unbalanced brace is confusing to both retrieval and the model. Treat a fenced code block as atomic and split around it.

The general principle: identify atomic units in your format and never split them. For a recipe corpus, an ingredient list is atomic. For documentation, a code example is. For a legal corpus, a numbered clause is. Deciding this per corpus is a half hour of work that outperforms most parameter tuning.

Metadata attached at chunk time

A chunk is not just text. Everything you know at index time and fail to record is lost, and reconstructing it later means reprocessing the corpus.

@dataclass(frozen=True)
class Chunk:
    id: str
    document_id: str
    text: str

    # Position
    index: int
    heading_path: list[str]
    page_number: int | None = None
    char_start: int | None = None

    # Provenance, carried from the Document record
    source_url: str | None = None
    source_domain: str | None = None
    ingested_at: datetime | None = None

    # Retrieval support
    embedding_model: str = ""
    token_count: int = 0

Each group earns its place. Position lets you fetch neighbouring chunks to expand context, and lets a citation point at a page rather than a document. Provenance is what makes a citation possible at all, which Lesson 5 covers. Retrieval support records which embedding model produced the vector, which the next lesson explains is not optional.

Note document_id on every chunk. The relationship from chunk back to document is what allows retrieving a small chunk and generating from a larger surrounding region, which is a pattern worth knowing: match precisely, generate with context.

Concept check. Your corpus is scraped recipe pages. Retrieval works well for ingredient questions and poorly for method questions. What would you check first?

Answer

Whether method sections are being split mid-step. Ingredient lists are short, self-contained, and full of distinctive nouns, so they survive almost any chunking and match queries well. Method sections are long, sequential, and full of pronouns referring to earlier steps, so they are exactly the content that fixed-size splitting damages.

Look at the actual chunks for a few recipes rather than at scores. Reading twenty chunks from your own corpus is the fastest diagnostic in this entire module, and it usually reveals the problem immediately.

Likely fixes: split the method on step boundaries rather than character count, keep each numbered step whole, and prepend the recipe title and section heading to every chunk so a step about baking is identifiable as belonging to focaccia.