Lesson 3: Search

Vector search, keyword search, and hybrid

Vector search finds chunks whose embeddings are closest to the query embedding. It captures meaning, so it matches paraphrases and related concepts without shared vocabulary.

It is weak on exactly the things that are unambiguous. Product codes, error identifiers, proper nouns, acronyms, and numbers all embed poorly, because the model has learned little about a token it rarely saw, and a rare exact term is precisely where a user expects an exact match.

Keyword search, usually BM25, ranks by term overlap with weighting that favours rare terms and shorter documents. It is exact, interpretable, and cheap, and it fails completely when the user and the document use different words for the same thing.

The two fail in complementary ways, which is why hybrid search is the standard production answer: run both and combine the rankings.

async def hybrid_search(query: str, *, top_k: int = 50) -> list[ScoredChunk]:
    """Run dense and sparse retrieval concurrently and fuse the rankings."""
    async with asyncio.TaskGroup() as group:
        dense_task = group.create_task(vector_search(query, top_k=top_k))
        sparse_task = group.create_task(keyword_search(query, top_k=top_k))

    return reciprocal_rank_fusion([dense_task.result(), sparse_task.result()], k=top_k)

The fan-out from Module 8, applied here: both searches run concurrently, so hybrid costs roughly the latency of the slower one rather than the sum.

Most vector databases now support hybrid natively, which removes the need to run a separate keyword index. Some embedding models also produce dense and sparse representations together, which is another way to get hybrid from one system.

Fusion strategies

Two ranked lists must become one, and the naive approach fails.

Score combination does not work directly. Cosine similarity ranges roughly zero to one and BM25 scores are unbounded and corpus-dependent, so adding them lets BM25 dominate arbitrarily. Normalising each list to a zero-to-one range helps but makes the result depend on the score distribution of that particular query, which is unstable.

Reciprocal Rank Fusion avoids the problem by using ranks rather than scores.

def reciprocal_rank_fusion(
    rankings: list[list[ScoredChunk]], *, k: int = 60, top_k: int = 50
) -> list[ScoredChunk]:
    """Combine ranked lists by rank position rather than by score."""
    scores: dict[str, float] = defaultdict(float)
    chunks: dict[str, ScoredChunk] = {}

    for ranking in rankings:
        for position, scored in enumerate(ranking, start=1):
            scores[scored.chunk.id] += 1.0 / (k + position)
            chunks[scored.chunk.id] = scored

    ranked = sorted(scores.items(), key=lambda item: item[1], reverse=True)
    return [replace(chunks[cid], score=score) for cid, score in ranked[:top_k]]

Each list contributes 1 / (k + rank), so a chunk ranked first in both lists scores higher than one ranked first in only one. The constant k, conventionally 60, dampens the difference between the top positions so that rank two is not dramatically worse than rank one.

RRF is scale-free, needs no tuning, and works when the two systems' scores are not comparable, which is why it became the default. Its limitation is that it ignores score magnitude entirely, so a chunk that matched overwhelmingly and one that matched marginally contribute identically if they hold the same rank.

Weighted combination is the alternative when you want to favour one retriever.

def weighted_fusion(dense, sparse, *, alpha: float = 0.7) -> list[ScoredChunk]:
    """alpha weights the dense ranking; requires normalised scores."""

The weight becomes a parameter to tune, which requires the evaluation set from Lesson 6. Without one, tuning it is guessing.

[IMAGE PROMPT M9-3
Purpose: Show how dense and sparse retrieval return different results and how reciprocal rank fusion combines them by position.
Visual type: Two-input fusion diagram with a worked ranking.
Prompt: A clean educational diagram with three columns. The left column is headed "Dense (vector)" and shows a ranked list of five chunk markers labelled top to bottom "C3", "C1", "C7", "C2", "C9", with a sub-label beneath reading "matches meaning, misses exact codes". The middle-left column is headed "Sparse (BM25)" and shows a different ranked list labelled "C7", "C4", "C3", "C9", "C1", with a sub-label reading "matches exact terms, misses paraphrase". Both columns feed arrows into a central box labelled "Reciprocal Rank Fusion" containing the formula text "score += 1 / (60 + rank)". From that box, an arrow leads to a right column headed "Fused" showing a ranked list labelled "C3", "C7", "C1", "C9", "C4", each with a small annotation showing which input lists contributed, for example "C3: dense 1, sparse 3". A note beneath the fused column reads "ranked well in both beats ranked first in one".
Required elements: Two differing input rankings with the same chunk identifiers appearing at different positions, a fusion box with the formula, a fused output ranking, per-item contribution annotations, the note.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, consistent list item sizing.
Layout: Three columns reading left to right with the fusion box centred between inputs and output.
Text labels: "Dense (vector)", "Sparse (BM25)", "Reciprocal Rank Fusion", "Fused", "C1", "C2", "C3", "C4", "C7", "C9", "score += 1 / (60 + rank)", "matches meaning, misses exact codes", "matches exact terms, misses paraphrase", "C3: dense 1, sparse 3", "ranked well in both beats ranked first in one".
Aspect ratio: 16:9
Accessibility: Identify list items by text label rather than colour, and show rank position explicitly by vertical order.
Avoid: Vendor logos, decorative icons, screenshots, tiny text, watermarks.
Alt text: Diagram showing a dense ranking and a sparse ranking of the same chunks in different orders, combined by reciprocal rank fusion into one list where items ranked well in both sources rise to the top.
END IMAGE PROMPT]

Metadata filtering, pre versus post

Users want scoped searches: only this domain, only recipes from the last year, only documents they are permitted to see. That last one is a correctness requirement rather than a convenience.

Post-filtering retrieves first and filters afterwards.

results = await vector_search(query, top_k=50)
filtered = [r for r in results if r.chunk.source_domain == domain]

Simple and quietly broken. If the domain holds five percent of the corpus, then of fifty retrieved chunks perhaps two survive the filter, and you asked for fifty. If the domain holds one percent, you frequently get zero results for a query that had perfectly good answers, and the system reports nothing found rather than reporting that it looked in the wrong place.

Pre-filtering restricts the search space first.

results = await vector_search(
    query, top_k=50, filter={"source_domain": domain, "ingested_after": cutoff}
)

You get fifty results from within the filtered set, which is what was asked for. Every serious vector store supports filtered search, and the implementations differ in how well they maintain recall when the filter is very selective, since an approximate index navigating a graph can struggle when most nodes are excluded.

The rules. Always pre-filter for correctness-critical filters, above all permissions, because post-filtering a permission check means the search saw documents the user cannot see and the result count leaks information about them. Pre-filter for selective filters, meaning anything excluding most of the corpus. Post-filtering is acceptable only for weak filters that exclude a small fraction, and only when you over-retrieve to compensate.

Test with a highly selective filter specifically. A system that behaves well filtering to fifty percent of the corpus can return nothing at one percent, and only the selective case reveals it.

Reranking with a cross-encoder

The embedding model compares a query and a chunk by embedding each separately and measuring distance. That is fast, because chunk vectors are computed once at index time, and it is approximate, because the model never sees the query and the chunk together.

A cross-encoder reranker takes the query and one chunk as a single input and scores their relevance directly. It is far more accurate and far more expensive, since it cannot precompute anything and must run once per candidate.

The pattern that uses both well is retrieve wide, rerank narrow.

async def search_and_rerank(query: str, *, retrieve: int = 50, keep: int = 8) -> list[ScoredChunk]:
    """Cheap retrieval for recall, expensive reranking for precision."""
    candidates = await hybrid_search(query, top_k=retrieve)
    if not candidates:
        return []
    reranked = await reranker.rank(query, [c.chunk.text for c in candidates])
    return [candidates[r.index] for r in reranked[:keep]]

Retrieval only needs to get the right chunk into the top fifty. The reranker decides the order of those fifty, and only the top eight reach the model.

Cost and latency. Reranking fifty candidates adds a model call, typically tens to low hundreds of milliseconds, and a per-candidate cost. The tradeoff is usually worth it, and a cheaper embedding model plus a reranker often beats a more expensive embedding model alone at lower total cost.

When to skip it. When latency budget is tight and quality is already acceptable, when candidates are few enough that ordering barely matters, or when your evaluation shows no improvement, which does happen and is worth knowing rather than assuming.

An LLM can rerank by scoring or ordering candidates directly. It is more flexible and considerably more expensive and slower than a dedicated cross-encoder, and it is worth considering when relevance depends on reasoning rather than on topical match.

top_k as a budget decision

top_k is usually left at whatever the tutorial used. It is a budget decision with three constraints pulling against each other.

More chunks means better recall. The right answer is more likely to be somewhere in twenty chunks than in three.

More chunks means more tokens. Twenty chunks of 500 tokens is 10,000 tokens per query, in every request, which is a cost and a context budget question that the next module addresses directly.

More chunks can mean worse answers. Beyond a point, additional chunks are noise, and there is a real effect where relevant information placed in the middle of a long context is used less reliably than information at the start or end. Adding marginal chunks can push a good one into that region.

Use two different numbers. Retrieve wide, at fifty or more, because retrieval recall is what limits the ceiling. Send narrow, at five to ten after reranking, because that is what the model reads. The distinction is easy to miss when one top_k is used for both.

Set the final number by measuring answer quality at several values on your evaluation set rather than by intuition, and expect the curve to flatten and then decline.