Naive Bayes
Gaussian mixture models modeled how data is generated without labels. Naive Bayes applies the same generative thinking to classification: model how each class generates its features, then invert with Bayes' rule. It powered the first generation of spam filters and remains a startlingly strong baseline for text.
From Bayes' rule to a classifier
For a class and features :
The joint likelihood is the hard part — with binary features it's a table of entries per class. The naive assumption collapses it: treat features as conditionally independent given the class,
Now we need only small one-dimensional distributions per class, each estimable by simple counting. Prediction is:
(Logs because multiplying ten thousand small probabilities underflows; sums of logs don't.)
A spam filter from counts
With word counts as features (multinomial Naive Bayes), training is literally counting:
- — fraction of training emails that are spam
- — occurrences of "viagra" in spam, over total words in spam
Each word in a new email votes with its log-likelihood ratio, and the votes add. A word common in spam and rare in ham pushes hard toward spam; neutral words push nowhere.
Compose an email yourself and watch the votes add up:
Naive Bayes: every word casts a vote
Toggle words to compose an email. Each present word votes with its smoothed log-likelihood ratio log P(w|spam)/P(w|ham) — rightward rose bars say spam, leftward teal bars say ham. The votes (plus the prior) simply add; the sign of the sum is the verdict. No interactions, no weights to train — just counting.
One fix is mandatory: a word never seen in spam would give and veto every other word with . Laplace smoothing adds a pseudo-count (usually 1) to every count:
Pick the flavor to match the features
Multinomial — word counts, the text workhorse. Bernoulli — binary present/absent features. Gaussian — continuous features, fitting a per-class mean and variance to each (the same per-dimension Gaussian estimation you saw in the EM algorithm, but with labels making it a one-shot count).
Why the wrong assumption still works
Words are not independent given the class — "machine" and "learning" travel together. So is badly calibrated: correlated evidence gets double-counted and the predicted probabilities saturate toward 0 or 1. But classification only needs the argmax to be right, and the ranking of classes survives a lot of double-counting. Wrong probabilities, right answers.
The trade-offs follow directly:
- Strengths: trains in one pass, handles huge sparse vocabularies, needs little data, naturally multiclass, easy to update online
- Weaknesses: overconfident probabilities (don't threshold them naively), blind to feature interactions, beaten by discriminative models (logistic regression, boosted trees) when training data is plentiful
Exercise
Without Laplace smoothing, what happens when a test email contains a word never seen in spam during training?
Need a hint?
One factor in the product becomes zero.
Key takeaways
- Naive Bayes = Bayes' rule + conditional independence:
- Training is counting; prediction is summing log-likelihood votes
- Laplace smoothing prevents a single unseen word from vetoing the entire prediction
- The independence assumption breaks calibration, not (usually) the argmax — great classifier, untrustworthy probabilities
Related concepts
- Bayesian Inference: the rule this classifier inverts
- Gaussian Discriminant Analysis: the other generative classifier, with a different assumption
- Logistic Regression: the discriminative model that overtakes it with enough data