Lesson 2: Embeddings
What an embedding is, operationally
An embedding is a list of numbers representing a piece of text, produced so that texts with similar meaning have vectors that are close together.
vector = await provider.embed(["Bake at 220C for 25 minutes"], model=EMBED_MODEL)
# [0.021, -0.113, 0.087, ...] typically 384 to 3072 numbers
You do not need to know how the model produces them. What you need to know is what the numbers do.
Closeness is the whole mechanism. Search works by embedding the query, then finding stored vectors closest to it. "How hot should I bake focaccia" and "Bake at 220C for 25 minutes" share almost no words, and their vectors are close because their meanings are related. That is what embeddings buy you over keyword search.
Closeness is not truth. Two vectors being close means the texts are similar in whatever way the model learned, which is usually topical similarity. "Bake at 220C" and "Bake at 180C" are extremely close and say different things. This is why retrieval finds candidates and the model reads them, and why a similarity score is not a confidence score.
Embeddings are opaque and fixed. You cannot inspect dimension 402 and learn anything. You cannot adjust a vector to mean something slightly different. The only lever you have is which model you use and what text you feed it.
Model choice
[VOLATILE: the embedding model landscape changes every few months. Specific model names, scores, and prices in this section will date quickly. Verify before publishing, and prefer teaching the selection criteria over the current answers.]
Six dimensions to the decision.
Retrieval quality is what matters, and the headline benchmark number is not it. Public leaderboards such as MTEB average across many task types including classification and clustering, and a model that leads the average can be middling at retrieval. Filter to the retrieval subset before comparing anything.
There is a second problem with leaderboards: the datasets are public, so models trained recently may have seen them. Treat published scores as a shortlist and nothing more.
Vector dimensions determine storage and search cost. A 3072-dimension vector uses eight times the storage of a 384-dimension one, and search is correspondingly slower. Some models support Matryoshka representation, meaning the vector can be truncated to fewer dimensions with modest quality loss, which is a genuinely useful lever: dropping from 3072 to 1024 typically cuts storage by two thirds for a small quality cost.
Cost covers both the initial embedding of the corpus and every query thereafter. Prices vary by more than an order of magnitude between providers, which matters when embedding two million documents.
Context length is the maximum text the model accepts. Most chunks are well under it, but a long structure-aware chunk can exceed a short limit, and text beyond it is silently truncated rather than rejected, which is a quiet way to lose the end of every long chunk.
Domain fit often beats general quality. A model trained mostly on web text may handle legal citations, medical codes, or product identifiers poorly. If your corpus has specialised vocabulary, test specifically on it.
Language coverage. A multilingual model is required for a multilingual corpus, and multilingual quality varies by language pair in ways an average score hides.
The decision procedure that actually works. Shortlist two or three models from a leaderboard's retrieval subset, then evaluate them on fifty to a hundred real queries from your own corpus with known correct answers. Building that set takes an afternoon and is more informative than every benchmark, which is why Lesson 6 treats it as a prerequisite rather than an optimisation.
Two practical notes. A cheaper embedding model plus a reranker frequently beats an expensive embedding model alone, and costs less overall, because the reranker fixes ordering while the embedder only needs to get candidates into the top fifty. And be wary of the model in every tutorial: some widely copied defaults are several years old and were chosen for being small and fast rather than good.
Normalization and distance metrics
Three ways to measure closeness between vectors.
Cosine similarity measures the angle, ignoring magnitude. Range is minus one to one, with one meaning identical direction. This is the usual choice for text.
Dot product multiplies element-wise and sums, so it accounts for both angle and magnitude.
Euclidean distance measures straight-line distance, where smaller means more similar.
Normalization scales a vector to length one. The important consequence: for normalized vectors, cosine similarity and dot product give the same ranking, and Euclidean distance ranks identically too. So if your vectors are normalized, the choice of metric does not change your results, and dot product is fastest.
import numpy as np
def normalize(vector: list[float]) -> list[float]:
array = np.array(vector, dtype=np.float32)
norm = np.linalg.norm(array)
return (array / norm).tolist() if norm > 0 else array.tolist()
Most providers return normalized vectors already. The rule that matters is use the metric your model was trained for, and be consistent: embedding documents with one metric and querying with another silently degrades results without any error.
Store the metric alongside the model name in your configuration, so that changing the model forces a decision about the metric rather than inheriting the old one.
Batching, caching, and the cost of re-embedding
Embedding two million chunks is a Module 8 problem, and the machinery is already built.
async def embed_corpus(chunks: Iterable[Chunk]) -> AsyncIterator[EmbeddedChunk]:
"""Embed in batches with bounded concurrency and a content-keyed cache."""
semaphore = asyncio.Semaphore(10)
async def embed_batch(batch: tuple[Chunk, ...]) -> list[EmbeddedChunk]:
uncached = [c for c in batch if await cache.get(cache_key(c)) is None]
if uncached:
async with semaphore:
await limiter.acquire(sum(c.token_count for c in uncached))
vectors = await provider.embed([c.text for c in uncached], model=EMBED_MODEL)
if len(vectors) != len(uncached):
raise OutputError(f"expected {len(uncached)} vectors, got {len(vectors)}")
for chunk, vector in zip(uncached, vectors, strict=True):
await cache.set(cache_key(chunk), vector)
return [EmbeddedChunk(c, await cache.get(cache_key(c))) for c in batch]
for batch in batched(chunks, 64):
for embedded in await embed_batch(batch):
yield embedded
def cache_key(chunk: Chunk) -> str:
"""Key on content and model, so a model change misses deliberately."""
digest = hashlib.sha256(chunk.text.encode("utf-8")).hexdigest()[:32]
return f"emb:{EMBED_MODEL}:{digest}"
The cache key includes the model name. This is not decoration: without it, a cache populated by one model would silently serve its vectors after you switched to another, which produces a corpus where some vectors are from one model and some from another, and search quality collapses in a way that is very hard to attribute.
The cost of re-embedding is the point of the caching. At two million chunks, embedding is a real bill and a real number of hours. Anything that avoids repeating it, whether a cache keyed on content or a checkpoint from Module 4, pays for itself immediately.
Embedding versioning and reindexing
Vectors from different models are not comparable. They live in different spaces, and a distance between them is meaningless rather than merely inaccurate.
The consequence is strict: changing the embedding model requires re-embedding the entire corpus. There is no incremental path, no migration, and no way to mix. A corpus half in one space and half in another returns results that look plausible and are effectively random for the mismatched half.
This makes the model choice more consequential than it first appears, and it makes recording the model version on every chunk mandatory rather than tidy.
@dataclass(frozen=True)
class IndexVersion:
embedding_model: str
dimensions: int
metric: Literal["cosine", "dot", "euclidean"]
normalized: bool
created_at: datetime
Store this alongside the index and check it at query time. A mismatch between the query model and the index model should raise loudly, not proceed.
Reindexing without downtime uses the blue-green pattern from Module 7's migration lesson.
- Create a new index alongside the existing one
- Embed the corpus into it with the new model, which takes hours
- Evaluate the new index against your labelled query set from Lesson 6
- Switch reads to the new index by configuration
- Keep the old index until you are confident, then delete it
Step three is the one people skip, and it is the only step that tells you whether the new model is actually better for your corpus rather than better on a public benchmark.
[IMAGE PROMPT M9-2
Purpose: Show why changing embedding models requires a full reindex and how blue-green reindexing avoids downtime and mixed spaces.
Visual type: Two-panel comparison with a staged migration flow.
Prompt: A clean educational diagram in two stacked panels. The upper panel is headed "Incremental update: broken" and shows a single index container holding a mix of vector markers, some tagged "model A" and some tagged "model B", with a query marker entering tagged "query: model B" and two result arrows, one labelled "matches correctly" pointing at a model B marker and one labelled "distance is meaningless" pointing at a model A marker. An annotation reads "different models, different spaces, results look plausible and are random". The lower panel is headed "Blue-green reindex" and shows five numbered stages left to right: stage 1 "index A live, serving queries" with a container marked "model A"; stage 2 "build index B" with a second container marked "model B" and an arrow from a corpus icon; stage 3 "evaluate index B on labelled queries" with a small checklist marker; stage 4 "switch reads by config" with an arrow moving a query marker from container A to container B; stage 5 "delete index A when confident". A caption beneath reads "no mixed spaces, no downtime, reversible until step 5".
Required elements: A mixed-space index with two model tags and contrasting result arrows in the upper panel, five numbered migration stages with two separate containers in the lower panel, the evaluation stage explicitly present, the caption.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector.
Layout: Two panels stacked vertically, the upper reading left to right in one scene, the lower reading left to right in five stages.
Text labels: "Incremental update: broken", "Blue-green reindex", "model A", "model B", "query: model B", "matches correctly", "distance is meaningless", "different models, different spaces, results look plausible and are random", "index A live, serving queries", "build index B", "evaluate index B on labelled queries", "switch reads by config", "delete index A when confident", "no mixed spaces, no downtime, reversible until step 5".
Aspect ratio: 16:9
Accessibility: Tag every vector marker with its model name in text rather than relying on colour, and number the migration stages.
Avoid: Vendor logos, decorative icons, screenshots, tiny text, watermarks.
Alt text: Diagram showing an index containing vectors from two different embedding models returning meaningless distances, against a five stage blue-green reindex that builds a second index, evaluates it, switches reads by configuration, and deletes the old index.
END IMAGE PROMPT]