Lesson 5: Context Assembly

Ordering chunks and position effects

You have eight reranked chunks. The order you place them in the prompt affects the answer.

Models attend unevenly across a long context. Information at the beginning and end is used more reliably than information in the middle, an effect commonly called lost in the middle. It is stronger with longer contexts and varies by model, and the practical implication is that the middle of a long list of chunks is the weakest position.

Three orderings, each defensible.

Relevance order, most relevant first. The obvious choice, and it puts your best chunk in the strongest position.

Relevance-outward, placing the top chunks at the beginning and end and the weaker ones in the middle. This deliberately exploits the position effect.

def order_outward(chunks: list[ScoredChunk]) -> list[ScoredChunk]:
    """Best chunks at the edges, weaker ones in the middle."""
    ordered: deque[ScoredChunk] = deque()
    for index, chunk in enumerate(chunks):          # already sorted by relevance
        if index % 2 == 0:
            ordered.appendleft(chunk)
        else:
            ordered.append(chunk)
    return list(ordered)

Document order, restoring the original sequence within each document. When chunks are sequential parts of one procedure, presenting step four before step two is confusing regardless of relevance scores. For a recipe method, this is clearly right.

The honest position: which ordering wins depends on your content and model, the differences are usually modest, and the way to decide is to measure on your evaluation set. The reason to know about it is that ordering is a variable, and leaving it unconsidered means you have chosen one by accident.

The strongest lever is fewer chunks. Position effects matter most when the context is long. Eight well-chosen chunks are usually better than twenty, and reranking is what lets you send eight.

Deduplication and near-duplicate collapse

Retrieved chunks overlap in three ways. Overlapping chunks from Lesson 1 share text by construction. Corpora contain genuinely duplicated documents, since the same recipe is republished across sites. And several chunks from one document may all be retrieved for the same query.

Each duplicate wastes a slot in your top-k and its tokens buy nothing.

def deduplicate(chunks: list[ScoredChunk], *, similarity_threshold: float = 0.92) -> list[ScoredChunk]:
    """Drop near-duplicates, keeping the highest scoring of each group."""
    kept: list[ScoredChunk] = []
    for candidate in chunks:                        # sorted by score, best first
        if any(is_near_duplicate(candidate, existing, similarity_threshold) for existing in kept):
            continue
        kept.append(candidate)
    return kept

Exact duplicates are caught by the content hash from Module 2. Near-duplicates need either a similarity comparison between the candidate vectors, which is cheap since you already have them, or a text-shingle comparison such as MinHash.

Diversity is the related idea. Maximal Marginal Relevance selects chunks balancing relevance against dissimilarity from what is already selected, so a query with several aspects gets coverage rather than eight variations of the same passage. It is worth reaching for when queries are broad, and unnecessary when they are narrow and factual.

One caution. Two chunks from the same document that are genuinely different sections are not duplicates, and collapsing on document identity rather than on content similarity discards useful context. Deduplicate on what the text says, not on where it came from.

Citation plumbing

A grounded answer must say where its claims came from, and that only works if source identity survives the whole pipeline.

The chain from Module 1 onward: the Document record carries source_url and source_path, each Chunk carries document_id and its position, the retrieved ScoredChunk carries the chunk, and the assembled context labels each chunk so the model can refer to it.

def format_context(chunks: list[ScoredChunk]) -> tuple[str, dict[str, ScoredChunk]]:
    """Label each chunk so the model can cite it, and return the lookup."""
    parts: list[str] = []
    lookup: dict[str, ScoredChunk] = {}

    for index, scored in enumerate(chunks, start=1):
        label = f"S{index}"
        lookup[label] = scored
        heading = " > ".join(scored.chunk.heading_path) if scored.chunk.heading_path else ""
        parts.append(
            f"<source id=\"{label}\" title=\"{heading}\">\n{scored.chunk.text}\n</source>"
        )

    return "\n\n".join(parts), lookup

Then instruct the model to cite by label, and validate the citations it returns.

class GroundedAnswer(BaseModel):
    text: str
    citations: list[str] = Field(min_length=1)


def resolve_citations(answer: GroundedAnswer, lookup: dict[str, ScoredChunk]) -> list[Source]:
    """Convert model citation labels into real sources, rejecting invented ones."""
    invalid = [c for c in answer.citations if c not in lookup]
    if invalid:
        raise OutputError(f"answer cited unknown sources: {invalid}")
    return [to_source(lookup[label]) for label in answer.citations]

That validation is the semantic check from Module 6. A model can produce a citation label that was never in the context, and an unvalidated citation is worse than none, because it looks like evidence.

Two refinements worth knowing. Use short opaque labels rather than URLs in the context, because a URL invites the model to reproduce or modify it and wastes tokens. And consider verifying that the cited chunk actually supports the claim, either with a second model call or an entailment check, which is the difference between a citation that exists and a citation that is correct.

Query transformation

The user's query is often not the best search string.

Rewriting turns a conversational turn into a standalone query. This is essential for multi-turn conversation, because "how long for that one?" embeds to nothing useful.

async def rewrite_for_search(history: list[Message], query: str) -> str:
    """Resolve pronouns and context into a self-contained search query."""
    return await provider.generate(
        [Message(role="system", content=REWRITE_SYSTEM),
         Message(role="user", content=format_history(history, query))],
        model=CHEAP_MODEL,
    ).text.strip()

Use a cheap model. This runs on every query and sits directly in the latency path before retrieval can even begin.

Expansion adds synonyms or related terms. Useful when users and documents use different vocabulary, such as a corpus written in professional terms and users writing casually.

Decomposition splits a compound question into parts, each retrieved separately.

# "What temperature for focaccia and how does it differ from ciabatta?"
# becomes two retrievals, fused

Necessary for multi-hop questions, where no single chunk contains the answer and it must be assembled from several.

Hypothetical document embedding embeds a generated hypothetical answer rather than the question, on the reasoning that an answer looks more like the documents you are searching than a question does. It sometimes helps noticeably and it adds a model call before retrieval.

The cost discipline that applies to all of them. Every transformation adds latency before the user sees anything, adds a cost per query, and adds a failure mode. Measure whether each one improves recall on your evaluation set, and remove the ones that do not. Cache transformed queries where the same questions recur.

When retrieval is the wrong tool

Retrieval is not the answer to every question about your data, and reaching for it by default produces systems that answer poorly and expensively.

Aggregation questions. "How many recipes use anchovies?" cannot be answered from retrieved chunks, because any top-k is a sample. This is a database query, and the right architecture routes it to SQL rather than to a vector search.

Exhaustive questions. "List every recipe that contains a nut allergen." Retrieval returns the most similar, not all matches, and the difference is a safety problem here rather than a quality one.

Structured filtering. "Recipes under 30 minutes." That is a metadata filter, and no embedding is needed.

Recency questions. "What was added this week?" That is a sort by timestamp.

Small corpora. If everything fits comfortably in the context window, retrieval adds machinery, latency, and a failure mode in exchange for nothing. The threshold moves as context windows grow and as cost falls, and it is worth recalculating rather than assuming.

Reasoning over the whole corpus. "What is the most common technique across all bread recipes?" requires reading everything, which is an offline analysis job rather than a query.

The routing decision belongs early. Classify the query and send it to retrieval, to SQL, to a filter, or to a tool. A system that routes well outperforms a system that retrieves for everything, and the routing itself can be a cheap model call or, often, a set of rules.