GloVe Training Procedure
Word2Vec uses local context windows. GloVe (Pennington et al., 2014) instead trains on global co-occurrence statistics: it factorizes a corpus-wide count matrix rather than streaming over windows.
The key insight: co-occurrence ratios
The ratio is informative about word relationships:
| Probe word | Ratio | ||
|---|---|---|---|
| solid | high | low | large |
| gas | low | high | small |
| water | high | high | ≈ 1 |
| fashion | low | low | ≈ 1 |
GloVe is designed so that differences of word vectors capture these ratios, which is exactly what makes analogies ("king − man + woman ≈ queen") work.
The full training procedure
-
Build the co-occurrence matrix over the corpus: counts how often word appears within a context window of word (typical window size: 10 words, symmetric). One pass over the corpus; the matrix is sparse.
-
Downweight frequent pairs with a capping function:
(common choice: ). This stops extremely common pairs ("the" co-occurring with everything) from dominating the loss, while ignoring pairs that never co-occur ( — those terms vanish entirely).
- Learn two embedding matrices (main) and (context) plus bias vectors by minimizing the weighted least-squares objective:
Each term pushes the dot product of two word vectors toward the log co-occurrence count — frequent companions get aligned vectors.
- Final embedding = (sum of both matrices for each word). The two matrices are symmetric in the objective, and summing them gives a small but consistent quality boost.
Why log counts
The objective targets the log of each co-occurrence count rather than the raw count:
Why log counts?
Co-occurrence counts span many orders of magnitude (Zipf's law). Taking the log compresses this range so the least-squares objective isn't dominated by the handful of extremely frequent pairs, and it makes vector differences correspond to count ratios: .
The weighting function: a numeric example
Two pairs, :
Frequent pair — "ice" and "solid" co-occur times:
If the current dot product , this pair contributes to the loss (weight 1.0, full penalty).
Rare pair — "ice" and "fashion" co-occur time:
Even if the predicted dot product is off by 1.0, this pair contributes only — roughly 11× less than the frequent pair. The weighting function deliberately deprioritizes rare co-occurrences whose counts are noisy.
GloVe versus Word2Vec
| Property | Word2Vec | GloVe |
|---|---|---|
| Training objective | Local context window | Global co-occurrence matrix |
| Computation | Sequential (online SGD) | Can be parallelized |
| Performance | Similar | Similar |
| Transfer | Good | Good |
In practice the two produce embeddings of comparable quality; GloVe's batch formulation parallelizes more naturally, while Word2Vec streams over arbitrarily large corpora without materializing a matrix.
Related concepts
- Word Embeddings: Word2Vec, FastText, and the embedding landscape
- Text Preprocessing: tokenization that feeds the co-occurrence counts