Matrix Factorization
Memory-based collaborative filtering compares whole rating vectors, which is slow and brittle on sparse data. Matrix factorization instead learns a compact explanation: every user and every item is a short vector of latent factors, and a rating is just their dot product. This single idea powered the winning Netflix Prize entries and remains the backbone of model-based recommendation.
Real-world examples: the Netflix Prize (2009), Spotify's and Amazon's classic CF pipelines, and the "embedding tables" inside modern deep recommenders.
The latent-factor model
Represent each user by a vector and each item by . The predicted rating is their dot product:
Stacking the vectors into matrices (users) and (items), the whole rating matrix is approximated by a low-rank product:
Each latent factor is an unnamed "taste" dimension (the model discovers them — maybe one ends up tracking "action-vs-romance"). A user's vector says how much they like each dimension; an item's says how much it expresses each. The viz shows how a low rank reconstructs a structured rating matrix:
Matrix factorization: low-rank reconstruction
The 6×6 rating matrix R (left) is genuinely rank-3. A model with rank k approximates it as U·Vᵀ — k latent factors per user and per item. Raise k: the reconstruction (right) approaches R and the RMSE falls to ~0 at k=3. A few latent 'tastes' reconstruct the whole matrix with far fewer numbers than storing it directly.
The connection to SVD and low-rank approximation
This is exactly the low-rank approximation from the linear-algebra course. If were fully observed, the best rank- approximation would be its truncated SVD (Eckart–Young). Matrix factorization is the same low-rank idea, with two practical twists for recommendation:
- The matrix is mostly unknown. Classic SVD needs a complete matrix; you can't take the SVD of something that's 99% missing. So we fit factors only to the observed entries.
- We don't need orthogonality. Any that reconstruct the observed ratings will do, so we solve a regression instead of a strict decomposition.
The training objective
Fit the factors by minimizing squared error on the observed ratings , with L2 regularization to prevent overfitting the sparse data:
Two standard optimizers:
- SGD: for each observed rating, take a gradient step on and . Simple, scales to huge sparse data, the Netflix-Prize favorite.
- ALS (Alternating Least Squares): fix and solve for in closed form (it's just ridge regression per user), then fix and solve for ; alternate. Each half is convex and parallelizes beautifully — the choice for implicit feedback at scale (Spark MLlib).
Biases matter as much as factors
Real ratings are dominated by who is rating and what is being rated. Production models add bias terms: , where is the global mean, a user's tendency to rate high/low, and an item's overall popularity. These biases often explain more variance than the latent factors themselves.
Why low rank works
A low rank encodes an assumption: a few hidden tastes explain most of the rating behavior. This (a) compresses ratings into parameters, (b) generalizes — predicting unrated entries by completing the pattern, which is the whole point, and (c) acts as a regularizer against memorizing noise. Too small a underfits (misses real structure); too large overfits the sparse observations — tune it on held-out ratings.
Limitations
- Cold start persists: a new user/item has no ratings, so no factor vector can be learned (fall back to content features or biases).
- Linearity: a dot product captures only linear factor interactions — deep recommenders (next lesson) add nonlinearity.
- Implicit feedback needs the confidence-weighted variant (weighted ALS), since unobserved ≠ negative.
Exercise
In matrix factorization, a user is a vector p_u ∈ ℝᵏ and an item is q_i ∈ ℝᵏ. How is the predicted rating computed?
Need a hint?
It's the simplest possible combination of two vectors that yields a scalar.
Exercise
Matrix factorization is the low-rank/SVD idea adapted to recommendation. What is the key reason you can't just take the SVD of the rating matrix directly?
Need a hint?
What fraction of the rating matrix do you actually observe?
Exercise
Alternating Least Squares (ALS) fixes the item factors Q and solves for the user factors P, then alternates. Why is each half-step attractive computationally?
Need a hint?
With one matrix fixed, what kind of problem is solving for the other?
Common mistakes
- Fitting to the zeros. Train on observed entries only; treating missing ratings as 0 biases the factors toward "everyone dislikes everything."
- No regularization. With sparse data, unregularized factors overfit instantly — keep the term.
- Skipping bias terms. Pure ignores user/item rating tendencies that dominate real data.
- Picking by training error. Training error always falls with larger ; choose (and ) on a held-out set.
Key takeaways
- Matrix factorization models each user and item as a length- latent-factor vector; a rating is their dot product, so is low-rank.
- It's the SVD/low-rank idea adapted to a mostly-missing matrix: fit factors to observed entries with regularization, via SGD or ALS.
- Bias terms () capture user/item tendencies and often explain more than the factors.
- Low rank compresses and generalizes (predicting unrated entries) but inherits cold start and is linear — motivating deep recommenders.
Related concepts
- SVD & Low-Rank Approximation: the exact low-rank theory matrix factorization adapts
- Deep & Two-Tower Recommenders: the neural generalization that scales to billions of items
- Regularization: the L2 penalty that keeps sparse-data factors from overfitting