GloVe Training ProcedureOpen in Colab

GloVe Training Procedure

Word2Vec uses local context windows. GloVe (Pennington et al., 2014) instead trains on global co-occurrence statistics: it factorizes a corpus-wide count matrix rather than streaming over windows.

The key insight: co-occurrence ratios

The ratio P(k∣ice) / P(k∣steam)P(k \mid \text{ice})\, /\, P(k \mid \text{steam}) is informative about word relationships:

Probe word kkP(k∣ice)P(k \mid \text{ice})P(k∣steam)P(k \mid \text{steam})Ratio
solidhighlowlarge
gaslowhighsmall
waterhighhigh≈ 1
fashionlowlow≈ 1

GloVe is designed so that differences of word vectors capture these ratios, which is exactly what makes analogies ("king − man + woman ≈ queen") work.

The full training procedure

  1. Build the co-occurrence matrix X∈RV×VX \in \mathbb{R}^{V \times V} over the corpus: XijX_{ij} counts how often word jj appears within a context window of word ii (typical window size: 10 words, symmetric). One pass over the corpus; the matrix is sparse.

  2. Downweight frequent pairs with a capping function:

f(Xij)=min⁡ ⁣(1,  (Xij/xmax)0.75)f(X_{ij}) = \min\!\left(1,\; (X_{ij}/x_\text{max})^{0.75}\right)

(common choice: xmax=100x_\text{max} = 100). This stops extremely common pairs ("the" co-occurring with everything) from dominating the loss, while ignoring pairs that never co-occur (f(0)=0f(0) = 0 — those terms vanish entirely).

  1. Learn two embedding matrices V\mathbf{V} (main) and V~\tilde{\mathbf{V}} (context) plus bias vectors by minimizing the weighted least-squares objective:
L=∑i,j=1Vf(Xij)(vi⊤v~j+bi+b~j−log⁡Xij)2\mathcal{L} = \sum_{i,j=1}^V f(X_{ij})\left(\mathbf{v}_i^\top \tilde{\mathbf{v}}_j + b_i + \tilde{b}_j - \log X_{ij}\right)^2

Each term pushes the dot product of two word vectors toward the log co-occurrence count — frequent companions get aligned vectors.

  1. Final embedding = vi+v~i\mathbf{v}_i + \tilde{\mathbf{v}}_i (sum of both matrices for each word). The two matrices are symmetric in the objective, and summing them gives a small but consistent quality boost.

Why log counts

The objective targets the log of each co-occurrence count rather than the raw count:

Why log counts?

Co-occurrence counts span many orders of magnitude (Zipf's law). Taking the log compresses this range so the least-squares objective isn't dominated by the handful of extremely frequent pairs, and it makes vector differences correspond to count ratios: va⊤v~k−vb⊤v~k≈log⁡XakXbk\mathbf{v}_a^\top\tilde{\mathbf{v}}_k - \mathbf{v}_b^\top\tilde{\mathbf{v}}_k \approx \log\frac{X_{ak}}{X_{bk}}.

The weighting function: a numeric example

Two pairs, xmax=100x_\text{max}=100:

Frequent pair — "ice" and "solid" co-occur Xij=100X_{ij}=100 times:

f(100)=min⁡ ⁣(1, (100/100)0.75)=1.0,log⁡Xij=log⁡100≈4.605f(100) = \min\!\bigl(1,\,(100/100)^{0.75}\bigr) = 1.0, \quad \log X_{ij} = \log 100 \approx 4.605

If the current dot product vi⊤v~j+bi+b~j=4.0\mathbf{v}_i^\top\tilde{\mathbf{v}}_j + b_i + \tilde{b}_j = 4.0, this pair contributes (4.0−4.605)2≈0.366(4.0 - 4.605)^2 \approx 0.366 to the loss (weight 1.0, full penalty).

Rare pair — "ice" and "fashion" co-occur Xij=1X_{ij}=1 time:

f(1)=(1/100)0.75=0.010.75≈0.032,log⁡Xij=0f(1) = (1/100)^{0.75} = 0.01^{0.75} \approx 0.032, \quad \log X_{ij} = 0

Even if the predicted dot product is off by 1.0, this pair contributes only 0.032×1.0≈0.0320.032 \times 1.0 \approx 0.032 — roughly 11× less than the frequent pair. The weighting function deliberately deprioritizes rare co-occurrences whose counts are noisy.

GloVe versus Word2Vec

PropertyWord2VecGloVe
Training objectiveLocal context windowGlobal co-occurrence matrix
ComputationSequential (online SGD)Can be parallelized
PerformanceSimilarSimilar
TransferGoodGood

In practice the two produce embeddings of comparable quality; GloVe's batch formulation parallelizes more naturally, while Word2Vec streams over arbitrarily large corpora without materializing a matrix.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Referenced by
GloVe Training ProcedureWord EmbeddingsText Preprocessing