Matrix FactorizationOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Matrix Factorization

Memory-based collaborative filtering compares whole rating vectors, which is slow and brittle on sparse data. Matrix factorization instead learns a compact explanation: every user and every item is a short vector of latent factors, and a rating is just their dot product. This single idea powered the winning Netflix Prize entries and remains the backbone of model-based recommendation.

Real-world examples: the Netflix Prize (2009), Spotify's and Amazon's classic CF pipelines, and the "embedding tables" inside modern deep recommenders.

The latent-factor model

Represent each user uu by a vector pu∈Rk\mathbf{p}_u \in \mathbb{R}^k and each item ii by qi∈Rk\mathbf{q}_i \in \mathbb{R}^k. The predicted rating is their dot product:

R^ui=pu⊤qi=∑f=1kpuf qif\hat{R}_{ui} = \mathbf{p}_u^{\top}\mathbf{q}_i = \sum_{f=1}^{k} p_{uf}\,q_{if}

Stacking the vectors into matrices P∈Rn×k\mathbf{P}\in\mathbb{R}^{n\times k} (users) and Q∈Rm×k\mathbf{Q}\in\mathbb{R}^{m\times k} (items), the whole rating matrix is approximated by a low-rank product:

R^=PQ⊤,k≪n,m\hat{\mathbf{R}} = \mathbf{P}\mathbf{Q}^{\top}, \qquad k \ll n, m

Each latent factor is an unnamed "taste" dimension (the model discovers them — maybe one ends up tracking "action-vs-romance"). A user's vector says how much they like each dimension; an item's says how much it expresses each. The viz shows how a low rank reconstructs a structured rating matrix:

Matrix factorization: low-rank reconstruction

R (true ratings)≈U·Vᵀreconstruction (rank 1)
true rank3
RMSE0.474
params stored12 vs 36

The 6×6 rating matrix R (left) is genuinely rank-3. A model with rank k approximates it as U·Vᵀ — k latent factors per user and per item. Raise k: the reconstruction (right) approaches R and the RMSE falls to ~0 at k=3. A few latent 'tastes' reconstruct the whole matrix with far fewer numbers than storing it directly.

The connection to SVD and low-rank approximation

This is exactly the low-rank approximation from the linear-algebra course. If R\mathbf{R} were fully observed, the best rank-kk approximation would be its truncated SVD R≈UkΣkVk⊤\mathbf{R} \approx \mathbf{U}_k\mathbf{\Sigma}_k\mathbf{V}_k^{\top} (Eckart–Young). Matrix factorization is the same low-rank idea, with two practical twists for recommendation:

  1. The matrix is mostly unknown. Classic SVD needs a complete matrix; you can't take the SVD of something that's 99% missing. So we fit factors only to the observed entries.
  2. We don't need orthogonality. Any P,Q\mathbf{P}, \mathbf{Q} that reconstruct the observed ratings will do, so we solve a regression instead of a strict decomposition.

The training objective

Fit the factors by minimizing squared error on the observed ratings K={(u,i):Rui known}\mathcal{K} = \{(u,i): R_{ui}\text{ known}\}, with L2 regularization to prevent overfitting the sparse data:

min⁡P,Q∑(u,i)∈K(Rui−pu⊤qi)2  +  λ(∥pu∥2+∥qi∥2)\min_{\mathbf{P},\mathbf{Q}} \sum_{(u,i)\in\mathcal{K}} \big(R_{ui} - \mathbf{p}_u^{\top}\mathbf{q}_i\big)^2 \;+\; \lambda\big(\lVert\mathbf{p}_u\rVert^2 + \lVert\mathbf{q}_i\rVert^2\big)

Two standard optimizers:

  • SGD: for each observed rating, take a gradient step on pu\mathbf{p}_u and qi\mathbf{q}_i. Simple, scales to huge sparse data, the Netflix-Prize favorite.
  • ALS (Alternating Least Squares): fix Q\mathbf{Q} and solve for P\mathbf{P} in closed form (it's just ridge regression per user), then fix P\mathbf{P} and solve for Q\mathbf{Q}; alternate. Each half is convex and parallelizes beautifully — the choice for implicit feedback at scale (Spark MLlib).

Biases matter as much as factors

Real ratings are dominated by who is rating and what is being rated. Production models add bias terms: R^ui=μ+bu+bi+pu⊤qi\hat{R}_{ui} = \mu + b_u + b_i + \mathbf{p}_u^{\top}\mathbf{q}_i, where μ\mu is the global mean, bub_u a user's tendency to rate high/low, and bib_i an item's overall popularity. These biases often explain more variance than the latent factors themselves.

Why low rank works

A low rank kk encodes an assumption: a few hidden tastes explain most of the rating behavior. This (a) compresses n×mn\times m ratings into (n+m)×k(n+m)\times k parameters, (b) generalizes — predicting unrated entries by completing the pattern, which is the whole point, and (c) acts as a regularizer against memorizing noise. Too small a kk underfits (misses real structure); too large overfits the sparse observations — tune it on held-out ratings.

Limitations

  • Cold start persists: a new user/item has no ratings, so no factor vector can be learned (fall back to content features or biases).
  • Linearity: a dot product captures only linear factor interactions — deep recommenders (next lesson) add nonlinearity.
  • Implicit feedback needs the confidence-weighted variant (weighted ALS), since unobserved ≠ negative.

Exercise

In matrix factorization, a user is a vector p_u ∈ ℝᵏ and an item is q_i ∈ ℝᵏ. How is the predicted rating computed?

Need a hint?

It's the simplest possible combination of two vectors that yields a scalar.

Exercise

Matrix factorization is the low-rank/SVD idea adapted to recommendation. What is the key reason you can't just take the SVD of the rating matrix directly?

Need a hint?

What fraction of the rating matrix do you actually observe?

Exercise

Alternating Least Squares (ALS) fixes the item factors Q and solves for the user factors P, then alternates. Why is each half-step attractive computationally?

Need a hint?

With one matrix fixed, what kind of problem is solving for the other?

Common mistakes

  1. Fitting to the zeros. Train on observed entries only; treating missing ratings as 0 biases the factors toward "everyone dislikes everything."
  2. No regularization. With sparse data, unregularized factors overfit instantly — keep the λ\lambda term.
  3. Skipping bias terms. Pure pu⊤qi\mathbf{p}_u^{\top}\mathbf{q}_i ignores user/item rating tendencies that dominate real data.
  4. Picking kk by training error. Training error always falls with larger kk; choose kk (and λ\lambda) on a held-out set.

Key takeaways

  • Matrix factorization models each user and item as a length-kk latent-factor vector; a rating is their dot product, so R^=PQ⊤\hat{\mathbf{R}} = \mathbf{P}\mathbf{Q}^{\top} is low-rank.
  • It's the SVD/low-rank idea adapted to a mostly-missing matrix: fit factors to observed entries with regularization, via SGD or ALS.
  • Bias terms (μ+bu+bi\mu + b_u + b_i) capture user/item tendencies and often explain more than the factors.
  • Low rank compresses and generalizes (predicting unrated entries) but inherits cold start and is linear — motivating deep recommenders.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Matrix FactorizationLinear Algebra for MLThe Recommendation ProblemSVD & Low-Rank Approximat…Deep & Two-Tower Recommen…Regularization: Ridge & L…