Lesson 4: Vector Stores

A vector database is just another Retriever

Module 7 established that a database sits behind a repository interface and business logic never imports a driver. A vector store is not an exception, and treating it as one is how teams end up unable to change it.

class Retriever(Protocol):
    """Finds relevant chunks. Independent of how they are stored."""

    async def search(
        self,
        query_vector: list[float],
        *,
        top_k: int,
        filters: dict[str, object] | None = None,
    ) -> list[ScoredChunk]: ...

    async def upsert(self, chunks: list[EmbeddedChunk]) -> None: ...

    async def delete_by_document(self, document_id: str) -> None: ...

Three notes on the design. delete_by_document exists because deletion is the operation people forget until a document is retracted and its chunks keep being retrieved. filters is part of the interface because Lesson 3 showed that pre-filtering must happen inside the search rather than after it. And the interface takes a vector rather than a query string, so that embedding stays a separate concern and the same retriever works with any embedding model.

class QdrantRetriever:
    def __init__(self, client: QdrantClient, collection: str) -> None:
        self._client = client
        self._collection = collection

    async def search(self, query_vector, *, top_k, filters=None) -> list[ScoredChunk]:
        response = await self._client.search(
            collection_name=self._collection,
            query_vector=query_vector,
            limit=top_k,
            query_filter=to_qdrant_filter(filters),
        )
        return [to_scored_chunk(point) for point in response]

[VOLATILE: client APIs differ across vector stores and change between versions. This is illustrative.]

An in-memory implementation for tests is a list and a loop, and it makes retrieval logic testable without running a database, exactly as Module 7's in-memory repository did.

Index types and the recall and latency tradeoff

Searching two million vectors exhaustively is accurate and slow. Approximate nearest neighbour indexes trade a little accuracy for a large speed gain.

Flat, meaning exhaustive search, compares the query against every vector. Perfect recall, linear cost. Correct below roughly a hundred thousand vectors, and a useful ground truth for measuring how much recall your approximate index is losing.

HNSW builds a navigable graph and walks it toward the query. It is the common default: fast, good recall, and memory-hungry because the graph is held in memory. Two parameters matter. Higher ef_construction builds a better graph more slowly. Higher ef_search explores more at query time, improving recall and increasing latency, and it can be raised per query, which lets you spend more on queries that matter.

IVF partitions vectors into clusters and searches only the nearest few. nprobe controls how many clusters are searched, trading recall for speed. It is more memory-efficient than HNSW and generally slightly worse at the same recall.

Quantization compresses vectors, typically from 32-bit floats to 8-bit integers or binary. Storage falls dramatically, recall falls somewhat, and a common pattern is to search the compressed vectors and rescore the top candidates with full precision.

The one thing to take from this. These are all points on the same curve: recall against latency against memory, choose two. The right response to a slow search is not automatically a different index but a decision about which of the three you are willing to spend.

Measure the recall you are losing. Build a flat index over a sample, run your evaluation queries against both, and compare. Teams often discover their approximate index is at 85 percent recall when they assumed 99, and that missing 15 percent is invisible without the comparison.

[IMAGE PROMPT M9-4
Purpose: Show that index choice is a three-way tradeoff between recall, latency, and memory rather than a single best option.
Visual type: Comparison chart with plotted positions and an annotated axis pair.
Prompt: A clean educational diagram with a two-axis plot occupying most of the frame. The horizontal axis is labelled "query latency, lower is better" and the vertical axis is labelled "recall, higher is better". Four labelled points are plotted: "Flat" positioned at high recall and high latency, marked "exact, 100% recall"; "HNSW" positioned at high recall and low latency, marked "high memory"; "IVF" positioned at slightly lower recall and low latency, marked "lower memory"; "HNSW + quantization" positioned at moderate recall and lowest latency, marked "lowest memory". A dashed curve passes through HNSW, IVF, and the quantized point labelled "the tradeoff frontier". To the right, a small three-item legend lists "recall", "latency", "memory" under a heading reading "choose two". A note at the bottom reads "measure your recall against a flat index on a sample".
Required elements: Two labelled axes with direction indicated, four labelled and positioned index points with their memory annotations, a frontier curve, the choose-two legend, the measurement note.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, plot gridlines light.
Layout: Plot occupying the left three quarters, legend and note on the right and bottom.
Text labels: "query latency, lower is better", "recall, higher is better", "Flat", "HNSW", "IVF", "HNSW + quantization", "exact, 100% recall", "high memory", "lower memory", "lowest memory", "the tradeoff frontier", "choose two", "recall", "latency", "memory", "measure your recall against a flat index on a sample".
Aspect ratio: 16:9
Accessibility: Label every plotted point with text and annotate memory in words rather than encoding it in colour or size alone.
Avoid: Vendor logos, precise benchmark numbers, decorative elements, tiny text, watermarks.
Alt text: Plot of recall against query latency showing flat search with perfect recall and high latency, HNSW with high recall and low latency at high memory cost, IVF slightly lower on both, and quantized HNSW fastest with lowest memory and moderate recall.
END IMAGE PROMPT]

Keeping the store swappable

The registry pattern from Module 7 applies unchanged.

retriever_registry = ProviderRegistry()
retriever_registry.register("qdrant", lambda s: QdrantRetriever(build_qdrant(s), s.collection))
retriever_registry.register("pgvector", lambda s: PgVectorRetriever(build_engine(s)))
retriever_registry.register("memory", lambda s: InMemoryRetriever())

retriever = retriever_registry.create(settings.retriever_name, settings)

What swapping actually requires, because the interface is necessary and not sufficient. Filter syntax differs between stores, so to_qdrant_filter is not portable and each implementation needs its own translation. Index configuration differs, and the tuning you did for one does not transfer. Hybrid search support differs, and a store without it means running a separate keyword index. And you must rebuild the index in the new store, since the vectors can be exported but the index structure and its tuning cannot.

The interface makes the change a bounded piece of work in known places rather than a rewrite. That is the realistic claim, and it is worth the effort.