Naive BayesOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Naive Bayes

Gaussian mixture models modeled how data is generated without labels. Naive Bayes applies the same generative thinking to classification: model how each class generates its features, then invert with Bayes' rule. It powered the first generation of spam filters and remains a startlingly strong baseline for text.

From Bayes' rule to a classifier

For a class yy and features x=(x1,…,xd)\mathbf{x} = (x_1, \ldots, x_d):

P(y∣x)∝P(y) P(x∣y)P(y \mid \mathbf{x}) \propto P(y) \, P(\mathbf{x} \mid y)

The joint likelihood P(x∣y)P(\mathbf{x} \mid y) is the hard part — with dd binary features it's a table of 2d2^d entries per class. The naive assumption collapses it: treat features as conditionally independent given the class,

P(x∣y)=∏j=1dP(xj∣y)P(\mathbf{x} \mid y) = \prod_{j=1}^{d} P(x_j \mid y)

Now we need only dd small one-dimensional distributions per class, each estimable by simple counting. Prediction is:

y^=arg⁡max⁡y  log⁡P(y)+∑jlog⁡P(xj∣y)\hat{y} = \arg\max_y \; \log P(y) + \sum_{j} \log P(x_j \mid y)

(Logs because multiplying ten thousand small probabilities underflows; sums of logs don't.)

A spam filter from counts

With word counts as features (multinomial Naive Bayes), training is literally counting:

  • P(spam)P(\text{spam}) — fraction of training emails that are spam
  • P("viagra"∣spam)P(\text{"viagra"} \mid \text{spam}) — occurrences of "viagra" in spam, over total words in spam

Each word in a new email votes with its log-likelihood ratio, and the votes add. A word common in spam and rare in ham pushes hard toward spam; neutral words push nowhere.

Compose an email yourself and watch the votes add up:

Naive Bayes: every word casts a vote

← votes hamvotes spam →prior P(spam)=0.4-0.41"free"+2.03"meeting"-2.30sum-0.68
total log-odds-0.68
P(spam | email)33.7%
verdict: HAM

Toggle words to compose an email. Each present word votes with its smoothed log-likelihood ratio log P(w|spam)/P(w|ham) — rightward rose bars say spam, leftward teal bars say ham. The votes (plus the prior) simply add; the sign of the sum is the verdict. No interactions, no weights to train — just counting.

One fix is mandatory: a word never seen in spam would give P(xj∣spam)=0P(x_j \mid \text{spam}) = 0 and veto every other word with log⁡0=−∞\log 0 = -\infty. Laplace smoothing adds a pseudo-count α\alpha (usually 1) to every count:

P(xj∣y)=count(xj,y)+αcount(y)+α dP(x_j \mid y) = \frac{\text{count}(x_j, y) + \alpha}{\text{count}(y) + \alpha \, d}

Pick the flavor to match the features

Multinomial — word counts, the text workhorse. Bernoulli — binary present/absent features. Gaussian — continuous features, fitting a per-class mean and variance to each (the same per-dimension Gaussian estimation you saw in the EM algorithm, but with labels making it a one-shot count).

Why the wrong assumption still works

Words are not independent given the class — "machine" and "learning" travel together. So P(y∣x)P(y \mid \mathbf{x}) is badly calibrated: correlated evidence gets double-counted and the predicted probabilities saturate toward 0 or 1. But classification only needs the argmax to be right, and the ranking of classes survives a lot of double-counting. Wrong probabilities, right answers.

The trade-offs follow directly:

  • Strengths: trains in one pass, handles huge sparse vocabularies, needs little data, naturally multiclass, easy to update online
  • Weaknesses: overconfident probabilities (don't threshold them naively), blind to feature interactions, beaten by discriminative models (logistic regression, boosted trees) when training data is plentiful

Exercise

Without Laplace smoothing, what happens when a test email contains a word never seen in spam during training?

Need a hint?

One factor in the product becomes zero.

Key takeaways

  • Naive Bayes = Bayes' rule + conditional independence: P(x∣y)=∏jP(xj∣y)P(\mathbf{x}|y) = \prod_j P(x_j|y)
  • Training is counting; prediction is summing log-likelihood votes
  • Laplace smoothing prevents a single unseen word from vetoing the entire prediction
  • The independence assumption breaks calibration, not (usually) the argmax — great classifier, untrustworthy probabilities

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Naive BayesClusteringPCA & Dimensionality Redu…Probability & StatisticsEntropy & KL DivergenceGeneralized Linear Models…Gaussian Mixture ModelsThe EM AlgorithmBayesian InferenceLogistic Regression