Embeddings & Semantic Search
Keyword search matches strings. Semantic search matches meaning: "How do I get my money back?" should find a document titled "Refund policy" even though they share no words. The enabling idea is the embedding — a vector representation of text where distance encodes semantic similarity. This lesson builds the retrieval engine that the next lesson (RAG) plugs an LLM into.
Real-world examples: document and code search, deduplication, recommendation, clustering, and the retrieval half of every RAG system.
From text to vectors
An embedding model maps a piece of text to a dense vector (typical = 384–3072). Trained well, it places paraphrases close together and unrelated text far apart. Modern sentence/document embedders are themselves transformer encoders fine-tuned with a contrastive objective — pull matching pairs together, push mismatched pairs apart.
Not the same as token embeddings
The token embeddings inside a transformer represent single tokens. A sentence/document embedding represents an entire passage as one vector — usually by pooling the model's contextual token outputs. Same idea (vectors that encode meaning), different granularity.
Measuring similarity
Once text is a vector, similarity is geometry. The workhorse is cosine similarity — the cosine of the angle between two vectors, ignoring their magnitude:
It ranges from (opposite) through (unrelated) to (identical direction). Cosine is preferred over raw Euclidean distance because it focuses on direction (topic) rather than length (which often just tracks text length).
Normalized embeddings: cosine = dot product
Modern embedding models (OpenAI, Sentence-BERT) output L2-normalized (unit-norm) vectors: . For unit vectors the denominator is , so exactly. This is why vector databases default to inner-product (dot-product) indexes — it's the same as cosine but cheaper. If you're unsure whether your embeddings are normalized, normalize first with a / np.linalg.norm(a).
import numpy as np
def cosine(a, b):
return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
query = embed("how do I get my money back?")
docs = [embed(d) for d in corpus]
ranked = sorted(corpus, key=lambda d: cosine(query, embed(d)), reverse=True)
Semantic vs keyword search
| Keyword (BM25 / TF-IDF) | Semantic (embeddings) | |
|---|---|---|
| Matches | exact terms | meaning / paraphrase |
| "money back" → "refund" | ✗ | ✓ |
| Rare exact IDs / codes | ✓ | sometimes ✗ |
| Needs a model | no | yes (embedder) |
Neither dominates everywhere — production systems often run hybrid search (combine both scores) to get exact-match precision and semantic recall.
Keyword, semantic, and the constant nobody tunes
Two rankings that disagree
Pick a query and compare the two columns. The paraphrase queries are where keyword search struggles — nothing in “where is my parcel” lexically matches “track your parcel while it is in transit” beyond one word.
Try 'guarantee claim' and then 'SKU-88421'. The winner swaps.
neither one dominates
across 7 queries: BM25 better on 1, dense on 3, tied on 3
BM25 is computed for real over these 15 documents. The dense retriever is constructed rather than downloaded — words sharing a sense share a direction, and every token containing a digit collapses onto one shared direction, which is what a subword embedder does to rare codes. The synonym table is an assumption; the ID collapse and every number below are consequences.
Two things are worth taking from measuring that rather than asserting it. Semantic search's failure on exact codes is structural, not a tuning problem. In the walkthrough the two near-identical SKU documents have a cosine similarity of 1.0000 — tokens containing digits collapse onto a shared direction, so the representation has no way to express the difference between SKU-88421 and SKU-88422. Asked for one, its top hit is a password-reset document. Information the encoding discarded cannot be recovered downstream, which is the real argument for keeping a keyword index alongside.
And fusion's constant is not a detail. Reciprocal-rank fusion scores by from each retriever; the everyone copies from the original paper is calibrated for corpora of millions, where tail-rank differences are noise. Over a small corpus it flattens exactly the rank differences fusion exists to exploit. At hybrid scores 0.989 mean and — the number that matters — 0.920 on its worst query, against the dense retriever's 0.631. Past the benefit switches off completely: 0.945 mean and 0.613 worst, below dense on both. Fusion's real product is that worst-query floor, and you only get it if you tune against your own corpus.
Scaling: the vector index
Comparing a query against millions of vectors one-by-one (exact nearest-neighbour) is too slow. Vector databases use Approximate Nearest Neighbour (ANN) indexes (e.g. HNSW graphs, IVF) that trade a little recall for orders-of-magnitude speed.
The ANN trade-off
ANN returns most of the true neighbours, most of the time. You tune a knob (graph degree, number of probes) that trades recall vs latency. For RAG this is usually fine, but be aware your retriever is approximate.
The retrieval step we just built is exactly what the next lesson visualises and feeds into an LLM.
For how those ANN indexes actually work (IVF, HNSW, product quantization, and tuning the recall/latency knob), see the deep dive:
Exercise
Why is cosine similarity, rather than raw keyword overlap, the basis of semantic search?
Need a hint?
Consider 'How do I get my money back?' vs 'refund policy'.
Common mistakes
- Mixing embedding models: a query embedded by model A can't be compared to documents embedded by model B.
- Forgetting to normalise when your similarity assumes unit vectors.
- Assuming ANN is exact: it isn't; validate recall on a sample.
- Using semantic search for exact-ID lookups: keyword/hybrid is better there.
Key takeaways
- An embedding turns text into a vector whose geometry encodes meaning.
- Cosine similarity ranks documents by semantic closeness to a query, matching paraphrases with no shared words.
- Hybrid search combines keyword precision with semantic recall.
- ANN vector indexes make search over millions of vectors fast, at the cost of approximate recall.
Related concepts
- Word Embeddings: where the "meaning as vectors" idea began
- Retrieval-Augmented Generation: feeding retrieved chunks to an LLM