Word Embeddings
One-hot vectors tell the model nothing about word meaning — "cat" and "feline" have orthogonal representations despite being synonyms. Dense word embeddings map each word to a point in a continuous vector space where semantically similar words are geometrically close.
Word2Vec
Word2Vec (Mikolov et al., 2013) learns embeddings by training a shallow neural network to predict context from words or words from context. The key insight: a word's meaning can be inferred from the company it keeps.
Skip-gram
Task: given a center word, predict its surrounding context words.
For the sentence "The quick brown fox jumps over the lazy dog":
- Center word: "fox", window size
- Predict: ["quick", "brown", "jumps", "over"]
The model maximizes:
with a softmax over the full vocabulary:
where is the input embedding and is the output embedding.
Negative sampling approximates the expensive softmax: instead of normalizing over words, sample random "noise" words and train a binary classifier:
CBOW (Continuous Bag of Words)
Task: given the context words, predict the center word. Cheaper to train than skip-gram; better for frequent words. Skip-gram is better for rare words.
| Model | Predicts | Better for |
|---|---|---|
| Skip-gram | Context from center | Rare words, small data |
| CBOW | Center from context | Frequent words, fast training |
Semantic analogies
The famous "king – man + woman = queen" analogy works because Word2Vec embeddings encode semantic relationships as vector offsets:
This holds for many relationships: Paris–France+Italy ≈ Rome, walked–walk+swim ≈ swam.
# Using gensim
from gensim.models import KeyedVectors
# wv = KeyedVectors.load_word2vec_format('GoogleNews-vectors.bin', binary=True)
# result = wv.most_similar(positive=['king', 'woman'], negative=['man'])
# print(result[0]) # ('queen', 0.712)
The offset is real; the benchmark is generous
The offset is real. The benchmark is generous.
A constructed 24-dimensional space: seven random attribute axes, each word the sum of the attributes it has plus isotropic noise, L2-normalised. Bars are cosine similarity to a − b + c. Words struck through are the query words, which every standard analogy benchmark removes from the candidate set before scoring.
The space here is constructed, not downloaded (seven random attribute axes, each word a weighted sum of the attributes it has, plus noise), so there is no question about whether the structure exists. It does, by construction, and king − man + woman lands on queen.
What is worth knowing is the fine print. Every standard analogy benchmark removes the three query words from the candidate set before scoring, and that rule is doing more work than the embeddings are. Toggle it off and sweep the noise:
| noise | scored correct (inputs excluded) | nearest vector is actually a query word |
|---|---|---|
| 0.10 | 7/7 | 0/7 |
| 0.22 | 7/7 | 3/7 |
| 0.40 | 7/7 | 4/7 |
| 0.60 | 6/7 | 6/7 |
| 0.90 | 5/7 | 7/7 |
At noise the benchmark still reports a perfect while the honest nearest neighbour to is one of the words you started from more than half the time. The offset exists and points the right way; it is simply shorter than the distance back to king. Read the analogy result as "the offset is in the right direction", not "the model can do analogies", which is the substance of Linzen's and Nissim et al.'s critiques of the task.
GloVe (Global Vectors)
Word2Vec uses local context windows. GloVe (Pennington et al., 2014) instead uses global co-occurrence statistics: it builds a corpus-wide count matrix of which words appear near which, then learns embeddings whose dot products match the log co-occurrence counts via weighted least squares. The key insight is that co-occurrence ratios carry meaning ( is large for and small for ) and GloVe's objective makes vector differences capture exactly these ratios.
In practice GloVe and Word2Vec produce embeddings of similar quality; GloVe's batch formulation parallelizes more naturally.
FastText
FastText (Bojanowski et al., 2017) extends Word2Vec by representing each word as a bag of character n-grams:
Key advantages:
- Handles OOV words: "unhappiness" can be represented via its n-grams even if not in the vocabulary
- Better for morphologically rich languages (German, Turkish, Finnish)
- Better at capturing suffixes and prefixes
# FastText handles unseen words
# ft_model.get_word_vector("unknownword") # returns vector from character n-grams
Contextual embeddings
Word2Vec, GloVe, and FastText produce a single vector per word regardless of context. "Bank" in "river bank" and "bank account" gets the same embedding.
Contextual embeddings (ELMo, BERT, GPT) produce different vectors depending on context:
The representation of "bank" at position depends on the full sentence.
ELMo (Peters et al., 2018): run a bidirectional LSTM over the entire sentence, combine hidden states from multiple layers. Used as features on top of task-specific models.
BERT (Devlin et al., 2018): Transformer encoder trained on masked language modeling. Every token's representation encodes its full sentence context.
The embedding revolution
Static embeddings (Word2Vec, GloVe) were state-of-the-art until ~2018. Contextual embeddings (ELMo, BERT) dramatically improved performance on nearly every NLP task by handling polysemy and long-range dependencies. Today, all large language models use contextual token representations.
Evaluation: semantic similarity
Embeddings are evaluated on tasks like WordSim-353 (human-judged similarity pairs) and the analogy task:
Exercise
In the Word2Vec skip-gram model, what is the training objective?
Need a hint?
Skip-gram: center → context. CBOW: context → center.
Exercise
Word2Vec embeddings satisfy v(king) − v(man) + v(woman) ≈ v(queen). What does this imply about the embedding space?
Need a hint?
What kind of structure allows subtraction and addition to capture semantic relationships?
Exercise
The word "bank" appears in two sentences: "She sat by the river bank" and "He opened a bank account". How do static embeddings (Word2Vec) and contextual embeddings (BERT) differ in representing this word?
Need a hint?
Static = one vector per word type. Contextual = one vector per word occurrence.
Common mistakes
- Using static embeddings for polysemous words: "bat" (animal) and "bat" (sports equipment) get identical vectors with Word2Vec; use contextual embeddings if this matters.
- Averaging word vectors for sentence similarity: averaging is order-blind, so "Dog bites man" and "Man bites dog" collapse to the exact same vector. Use sentence-level models for similarity tasks.
- Not normalizing vectors before cosine similarity: unnormalized dot products mix magnitude with direction.
- Training word vectors on too little data: Word2Vec embeddings need >1M words to produce meaningful geometry. Use pre-trained vectors otherwise.
- Forgetting to align embedding spaces when combining: GloVe-300d and Word2Vec-300d live in different vector spaces; you can't concatenate and compare across models without alignment.
Key takeaways
- Word2Vec trains a neural network to predict context (skip-gram) or center word (CBOW); the hidden-layer weights are the embeddings.
- Skip-gram uses negative sampling to approximate the full softmax; it's better for rare words.
- GloVe factorizes the global co-occurrence matrix; similar quality to Word2Vec, more parallelizable.
- FastText uses character n-grams, handling OOV and morphologically rich languages.
- Contextual embeddings (BERT, GPT) produce different vectors per word per context, solving polysemy and enabling transfer learning.
Related concepts
- Text Preprocessing: tokenization that feeds into embeddings
- Sequence Models to BERT: how Transformers produce contextual embeddings