Embeddings & Semantic SearchOpen in Colab
Agentic project loop
Task definition·Context & tools·Orchestration loop·Evaluation·Guardrails·Operations feedback

Embeddings & Semantic Search

Keyword search matches strings. Semantic search matches meaning: "How do I get my money back?" should find a document titled "Refund policy" even though they share no words. The enabling idea is the embedding — a vector representation of text where distance encodes semantic similarity. This lesson builds the retrieval engine that the next lesson (RAG) plugs an LLM into.

Real-world examples: document and code search, deduplication, recommendation, clustering, and the retrieval half of every RAG system.

From text to vectors

An embedding model maps a piece of text to a dense vector e∈Rd\mathbf{e} \in \mathbb{R}^d (typical dd = 384–3072). Trained well, it places paraphrases close together and unrelated text far apart. Modern sentence/document embedders are themselves transformer encoders fine-tuned with a contrastive objective — pull matching pairs together, push mismatched pairs apart.

Not the same as token embeddings

The token embeddings inside a transformer represent single tokens. A sentence/document embedding represents an entire passage as one vector — usually by pooling the model's contextual token outputs. Same idea (vectors that encode meaning), different granularity.

Measuring similarity

Once text is a vector, similarity is geometry. The workhorse is cosine similarity — the cosine of the angle between two vectors, ignoring their magnitude:

cos(a,b)=a⋅b∥a∥ ∥b∥\text{cos}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a}\rVert\, \lVert \mathbf{b}\rVert}

It ranges from −1-1 (opposite) through 00 (unrelated) to 11 (identical direction). Cosine is preferred over raw Euclidean distance because it focuses on direction (topic) rather than length (which often just tracks text length).

Normalized embeddings: cosine = dot product

Modern embedding models (OpenAI, Sentence-BERT) output L2-normalized (unit-norm) vectors: ∥a∥=∥b∥=1\|\mathbf{a}\| = \|\mathbf{b}\| = 1. For unit vectors the denominator is 1×1=11 \times 1 = 1, so cos(a,b)=a⋅b\text{cos}(\mathbf{a}, \mathbf{b}) = \mathbf{a} \cdot \mathbf{b} exactly. This is why vector databases default to inner-product (dot-product) indexes — it's the same as cosine but cheaper. If you're unsure whether your embeddings are normalized, normalize first with a / np.linalg.norm(a).

Python
import numpy as np

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

query = embed("how do I get my money back?")
docs  = [embed(d) for d in corpus]
ranked = sorted(corpus, key=lambda d: cosine(query, embed(d)), reverse=True)

Semantic vs keyword search

Keyword (BM25 / TF-IDF)Semantic (embeddings)
Matchesexact termsmeaning / paraphrase
"money back" → "refund"✗✓
Rare exact IDs / codes✓sometimes ✗
Needs a modelnoyes (embedder)

Neither dominates everywhere — production systems often run hybrid search (combine both scores) to get exact-match precision and semantic recall.

Keyword, semantic, and the constant nobody tunes

Two retrieversstep 1 of 4
Two rankings that disagree
BM25 scores by term overlap; the dense retriever scores by direction in an embedding space. Run both over the same 15 documents and they return different orders for most queries.

Pick a query and compare the two columns. The paraphrase queries are where keyword search struggles — nothing in “where is my parcel” lexically matches “track your parcel while it is in transit” beyond one word.

Try 'guarantee claim' and then 'SKU-88421'. The winner swaps.

the same query, twice“guarantee claim” · paraphrase
BM25 — term overlap
1Claim a repair under the guarantee
2Refunds are issued within 30 days of purchase
3How to request your money back after a purchase
4Return an item and get a refund to your card
5Shipping options and delivery estimates
nDCG@3 0.613
dense — embedding cosine
1Claim a repair under the guarantee
2Warranty covers manufacturing defects for two years
3Refunds are issued within 30 days of purchase
4Bulk order pricing for wholesale buyers
5Free delivery on orders over fifty dollars
nDCG@3 1.000
neither one dominates

across 7 queries: BM25 better on 1, dense on 3, tied on 3

BM25denserelevant / hybrid

BM25 is computed for real over these 15 documents. The dense retriever is constructed rather than downloaded — words sharing a sense share a direction, and every token containing a digit collapses onto one shared direction, which is what a subword embedder does to rare codes. The synonym table is an assumption; the ID collapse and every number below are consequences.

Two things are worth taking from measuring that rather than asserting it. Semantic search's failure on exact codes is structural, not a tuning problem. In the walkthrough the two near-identical SKU documents have a cosine similarity of 1.0000 — tokens containing digits collapse onto a shared direction, so the representation has no way to express the difference between SKU-88421 and SKU-88422. Asked for one, its top hit is a password-reset document. Information the encoding discarded cannot be recovered downstream, which is the real argument for keeping a keyword index alongside.

And fusion's constant is not a detail. Reciprocal-rank fusion scores by 1/(k+rank)1/(k + \text{rank}) from each retriever; the k=60k = 60 everyone copies from the original paper is calibrated for corpora of millions, where tail-rank differences are noise. Over a small corpus it flattens exactly the rank differences fusion exists to exploit. At kle10k \\le 10 hybrid scores 0.989 mean and — the number that matters — 0.920 on its worst query, against the dense retriever's 0.631. Past k=20k = 20 the benefit switches off completely: 0.945 mean and 0.613 worst, below dense on both. Fusion's real product is that worst-query floor, and you only get it if you tune kk against your own corpus.

Scaling: the vector index

Comparing a query against millions of vectors one-by-one (exact nearest-neighbour) is too slow. Vector databases use Approximate Nearest Neighbour (ANN) indexes (e.g. HNSW graphs, IVF) that trade a little recall for orders-of-magnitude speed.

The ANN trade-off

ANN returns most of the true neighbours, most of the time. You tune a knob (graph degree, number of probes) that trades recall vs latency. For RAG this is usually fine, but be aware your retriever is approximate.

The retrieval step we just built is exactly what the next lesson visualises and feeds into an LLM.

For how those ANN indexes actually work (IVF, HNSW, product quantization, and tuning the recall/latency knob), see the deep dive:

Deep dive
Vector Databases & ANN Indexes
Includes runnable notebook
Deep dive
HNSW: Hierarchical Navigable Small World Graphs
Includes runnable notebook

Exercise

Why is cosine similarity, rather than raw keyword overlap, the basis of semantic search?

Need a hint?

Consider 'How do I get my money back?' vs 'refund policy'.

Common mistakes

  1. Mixing embedding models: a query embedded by model A can't be compared to documents embedded by model B.
  2. Forgetting to normalise when your similarity assumes unit vectors.
  3. Assuming ANN is exact: it isn't; validate recall on a sample.
  4. Using semantic search for exact-ID lookups: keyword/hybrid is better there.

Key takeaways

  • An embedding turns text into a vector whose geometry encodes meaning.
  • Cosine similarity ranks documents by semantic closeness to a query, matching paraphrases with no shared words.
  • Hybrid search combines keyword precision with semantic recall.
  • ANN vector indexes make search over millions of vectors fast, at the cost of approximate recall.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Embeddings & Semantic SearchAttention & TransformersNatural Language Processi…LLM Model TaxonomyTopic Modeling with BERTo…Training Embedding ModelsVision-Language ModelsWord EmbeddingsRetrieval-Augmented Gener…Vector Databases & ANN In…HNSW: Hierarchical Naviga…BM25: Lexical Ranking