Word EmbeddingsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Word Embeddings

One-hot vectors tell the model nothing about word meaning — "cat" and "feline" have orthogonal representations despite being synonyms. Dense word embeddings map each word to a point in a continuous vector space where semantically similar words are geometrically close.

Word2Vec

Word2Vec (Mikolov et al., 2013) learns embeddings by training a shallow neural network to predict context from words or words from context. The key insight: a word's meaning can be inferred from the company it keeps.

Skip-gram

Task: given a center word, predict its surrounding context words.

For the sentence "The quick brown fox jumps over the lazy dog":

  • Center word: "fox", window size w=2w=2
  • Predict: ["quick", "brown", "jumps", "over"]

The model maximizes:

L=1T∑t=1T∑−w≤j≤w,j≠0log⁡p(wt+j∣wt)\mathcal{L} = \frac{1}{T}\sum_{t=1}^T \sum_{-w \leq j \leq w, j \neq 0} \log p(w_{t+j} \mid w_t)

with a softmax over the full vocabulary:

p(wO∣wI)=exp⁡(vwO′⊤vwI)∑w=1Vexp⁡(vw′⊤vwI)p(w_O \mid w_I) = \frac{\exp(\mathbf{v}_{w_O}^{\prime\top} \mathbf{v}_{w_I})}{\sum_{w=1}^V \exp(\mathbf{v}_w^{\prime\top} \mathbf{v}_{w_I})}

where vw\mathbf{v}_w is the input embedding and vw′\mathbf{v}_w' is the output embedding.

Negative sampling approximates the expensive softmax: instead of normalizing over ∣V∣|V| words, sample kk random "noise" words and train a binary classifier:

LNS=log⁡σ(vwO′⊤vwI)+∑i=1kEwi∼Pn[log⁡σ(−vwi′⊤vwI)]\mathcal{L}_{\text{NS}} = \log \sigma(\mathbf{v}_{w_O}^{\prime\top} \mathbf{v}_{w_I}) + \sum_{i=1}^k \mathbb{E}_{w_i \sim P_n}\left[\log \sigma(-\mathbf{v}_{w_i}^{\prime\top} \mathbf{v}_{w_I})\right]

CBOW (Continuous Bag of Words)

Task: given the context words, predict the center word. Cheaper to train than skip-gram; better for frequent words. Skip-gram is better for rare words.

ModelPredictsBetter for
Skip-gramContext from centerRare words, small data
CBOWCenter from contextFrequent words, fast training

Semantic analogies

The famous "king – man + woman = queen" analogy works because Word2Vec embeddings encode semantic relationships as vector offsets:

vking−vman+vwoman≈vqueen\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}} \approx \mathbf{v}_{\text{queen}}

This holds for many relationships: Paris–France+Italy ≈ Rome, walked–walk+swim ≈ swam.

Python
# Using gensim
from gensim.models import KeyedVectors

# wv = KeyedVectors.load_word2vec_format('GoogleNews-vectors.bin', binary=True)
# result = wv.most_similar(positive=['king', 'woman'], negative=['man'])
# print(result[0])  # ('queen', 0.712)

The offset is real; the benchmark is generous

The offset is real. The benchmark is generous.

queen
0.907
princess
0.777
actress
0.639
nurse
0.458
girl
0.375
doctor
0.308
dogs
0.096
cats
-0.038
top hitqueen
expectedqueen
benchmark score, all 77/7
nearest vector is a query word4/7

A constructed 24-dimensional space: seven random attribute axes, each word the sum of the attributes it has plus isotropic noise, L2-normalised. Bars are cosine similarity to a − b + c. Words struck through are the query words, which every standard analogy benchmark removes from the candidate set before scoring.

The space here is constructed, not downloaded (seven random attribute axes, each word a weighted sum of the attributes it has, plus noise), so there is no question about whether the structure exists. It does, by construction, and king − man + woman lands on queen.

What is worth knowing is the fine print. Every standard analogy benchmark removes the three query words from the candidate set before scoring, and that rule is doing more work than the embeddings are. Toggle it off and sweep the noise:

noisescored correct (inputs excluded)nearest vector is actually a query word
0.107/70/7
0.227/73/7
0.407/74/7
0.606/76/7
0.905/77/7

At noise 0.400.40 the benchmark still reports a perfect 7/77/7 while the honest nearest neighbour to a−b+ca - b + c is one of the words you started from more than half the time. The offset exists and points the right way; it is simply shorter than the distance back to king. Read the analogy result as "the offset is in the right direction", not "the model can do analogies", which is the substance of Linzen's and Nissim et al.'s critiques of the task.

GloVe (Global Vectors)

Word2Vec uses local context windows. GloVe (Pennington et al., 2014) instead uses global co-occurrence statistics: it builds a corpus-wide count matrix of which words appear near which, then learns embeddings whose dot products match the log co-occurrence counts via weighted least squares. The key insight is that co-occurrence ratios carry meaning (P(k∣ice)/P(k∣steam)P(k \mid \text{ice}) / P(k \mid \text{steam}) is large for k=solidk = \text{solid} and small for k=gask = \text{gas}) and GloVe's objective makes vector differences capture exactly these ratios.

In practice GloVe and Word2Vec produce embeddings of similar quality; GloVe's batch formulation parallelizes more naturally.

Deep dive
GloVe Training Procedure
Includes runnable notebook

FastText

FastText (Bojanowski et al., 2017) extends Word2Vec by representing each word as a bag of character n-grams:

vwhere=vwh+vwhe+vher+vere+…\mathbf{v}_{\text{where}} = \mathbf{v}_{\text{wh}} + \mathbf{v}_{\text{whe}} + \mathbf{v}_{\text{her}} + \mathbf{v}_{\text{ere}} + \ldots

Key advantages:

  • Handles OOV words: "unhappiness" can be represented via its n-grams even if not in the vocabulary
  • Better for morphologically rich languages (German, Turkish, Finnish)
  • Better at capturing suffixes and prefixes
Python
# FastText handles unseen words
# ft_model.get_word_vector("unknownword")  # returns vector from character n-grams

Contextual embeddings

Word2Vec, GloVe, and FastText produce a single vector per word regardless of context. "Bank" in "river bank" and "bank account" gets the same embedding.

Contextual embeddings (ELMo, BERT, GPT) produce different vectors depending on context:

ht=f(x1,…,xT)[t]\mathbf{h}_t = f(\mathbf{x}_1, \ldots, \mathbf{x}_T)[t]

The representation of "bank" at position tt depends on the full sentence.

ELMo (Peters et al., 2018): run a bidirectional LSTM over the entire sentence, combine hidden states from multiple layers. Used as features on top of task-specific models.

BERT (Devlin et al., 2018): Transformer encoder trained on masked language modeling. Every token's representation encodes its full sentence context.

The embedding revolution

Static embeddings (Word2Vec, GloVe) were state-of-the-art until ~2018. Contextual embeddings (ELMo, BERT) dramatically improved performance on nearly every NLP task by handling polysemy and long-range dependencies. Today, all large language models use contextual token representations.

Evaluation: semantic similarity

Embeddings are evaluated on tasks like WordSim-353 (human-judged similarity pairs) and the analogy task:

sim(va,vb)=va⋅vb∥va∥∥vb∥\text{sim}(\mathbf{v}_a, \mathbf{v}_b) = \frac{\mathbf{v}_a \cdot \mathbf{v}_b}{\|\mathbf{v}_a\| \|\mathbf{v}_b\|}

Exercise

In the Word2Vec skip-gram model, what is the training objective?

Need a hint?

Skip-gram: center → context. CBOW: context → center.

Exercise

Word2Vec embeddings satisfy v(king) − v(man) + v(woman) ≈ v(queen). What does this imply about the embedding space?

Need a hint?

What kind of structure allows subtraction and addition to capture semantic relationships?

Exercise

The word "bank" appears in two sentences: "She sat by the river bank" and "He opened a bank account". How do static embeddings (Word2Vec) and contextual embeddings (BERT) differ in representing this word?

Need a hint?

Static = one vector per word type. Contextual = one vector per word occurrence.

Common mistakes

  1. Using static embeddings for polysemous words: "bat" (animal) and "bat" (sports equipment) get identical vectors with Word2Vec; use contextual embeddings if this matters.
  2. Averaging word vectors for sentence similarity: averaging is order-blind, so "Dog bites man" and "Man bites dog" collapse to the exact same vector. Use sentence-level models for similarity tasks.
  3. Not normalizing vectors before cosine similarity: unnormalized dot products mix magnitude with direction.
  4. Training word vectors on too little data: Word2Vec embeddings need >1M words to produce meaningful geometry. Use pre-trained vectors otherwise.
  5. Forgetting to align embedding spaces when combining: GloVe-300d and Word2Vec-300d live in different vector spaces; you can't concatenate and compare across models without alignment.

Key takeaways

  • Word2Vec trains a neural network to predict context (skip-gram) or center word (CBOW); the hidden-layer weights are the embeddings.
  • Skip-gram uses negative sampling to approximate the full softmax; it's better for rare words.
  • GloVe factorizes the global co-occurrence matrix; similar quality to Word2Vec, more parallelizable.
  • FastText uses character n-grams, handling OOV and morphologically rich languages.
  • Contextual embeddings (BERT, GPT) produce different vectors per word per context, solving polysemy and enabling transfer learning.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Word EmbeddingsRecurrent Neural NetworksAttention & TransformersText PreprocessingSequence Models to BERTEmbeddings & Semantic Sea…GloVe Training ProcedureByte-Pair Encoding (BPE)