Maximum Margin ClassificationOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Maximum Margin Classification (SVMs)

Support Vector Machines find the separating hyperplane with the widest possible margin — the "street" between classes. Wider streets generalize better. That "maximize the margin" rule is the loop's objective slot made unusually vivid: it's a precise definition of what a good boundary is, and the whole method follows from it.

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: Text classification (spam detection), bioinformatics (protein classification), handwriting recognition, image classification.

The margin

The margin is the distance between the decision boundary and the nearest data points from each class:

margin=2∥w∥\text{margin} = \frac{2}{\|\mathbf{w}\|}

Where does 2/∥w∥2/\|\mathbf{w}\| come from?

The signed distance from a point x\mathbf{x} to the hyperplane w⊤x+b=0\mathbf{w}^\top\mathbf{x}+b=0 is w⊤x+b∥w∥\dfrac{\mathbf{w}^\top\mathbf{x}+b}{\|\mathbf{w}\|} (project onto the unit normal w/∥w∥\mathbf{w}/\|\mathbf{w}\|). We are free to rescale w,b\mathbf{w},b so that the closest points satisfy w⊤x+b=±1\mathbf{w}^\top\mathbf{x}+b=\pm 1 — this fixes the otherwise-arbitrary scale. Then:

  • the +1+1 margin hyperplane is w⊤x+b=+1\mathbf{w}^\top\mathbf{x}+b=+1,
  • the −1-1 margin hyperplane is w⊤x+b=−1\mathbf{w}^\top\mathbf{x}+b=-1.

The distance from the boundary to either is ∣±1∣∥w∥=1∥w∥\dfrac{|\pm 1|}{\|\mathbf{w}\|}=\dfrac{1}{\|\mathbf{w}\|}, so the full street width is 2∥w∥\dfrac{2}{\|\mathbf{w}\|}. Maximizing the margin therefore means minimizing ∥w∥\|\mathbf{w}\| — equivalently 12∥w∥2\tfrac12\|\mathbf{w}\|^2, which is smooth. The SVM solves this minimization subject to every point being correctly classified with room to spare:

min⁡w,b12∥w∥2subject toyi(wTxi+b)≥1∀i\min_{\mathbf{w}, b} \frac{1}{2} \|\mathbf{w}\|^2 \quad \text{subject to} \quad y_i(\mathbf{w}^T \mathbf{x}_i + b) \geq 1 \quad \forall i

Intuition

Imagine you're placing a street between two neighborhoods. The wider the street, the more confident you are that new houses will be on the correct side. SVM finds the widest possible street.

Worked example (solved by hand)

Two classes along the line y=xy=x:

  • Class +1+1: (2,2), (3,3), (4,4)(2,2),\,(3,3),\,(4,4)
  • Class −1-1: (0,0), (−1,−1), (−2,−2)(0,0),\,(-1,-1),\,(-2,-2)

The closest opposing pair is (2,2)(2,2) and (0,0)(0,0) — these are the support vectors. By symmetry the boundary is their perpendicular bisector, with normal w∝(2,2)−(0,0)=(1,1)\mathbf{w}\propto(2,2)-(0,0)=(1,1). Write w=a(1,1)\mathbf{w}=a(1,1) and impose the margin equations on the two support vectors:

w⊤(2,2)+b=+1:    4a+b=1,w⊤(0,0)+b=−1:    b=−1.\mathbf{w}^\top(2,2)+b = +1:\;\; 4a+b=1, \qquad \mathbf{w}^\top(0,0)+b = -1:\;\; b=-1.

Solving: b=−1b=-1, then 4a=2⇒a=0.54a=2 \Rightarrow a=0.5, so w=(0.5, 0.5)\mathbf{w}=(0.5,\,0.5), b=−1b=-1. The boundary is 0.5x+0.5y−1=00.5x+0.5y-1=0, i.e. x+y=2x+y=2.

Margin: ∥w∥=0.52+0.52=0.5≈0.707\|\mathbf{w}\| = \sqrt{0.5^2+0.5^2}=\sqrt{0.5}\approx 0.707, so

margin=2∥w∥=20.707=22≈2.83,\text{margin} = \frac{2}{\|\mathbf{w}\|} = \frac{2}{0.707} = 2\sqrt{2}\approx 2.83,

which equals the distance between the two support vectors (2−0)2+(2−0)2=8=22\sqrt{(2{-}0)^2+(2{-}0)^2}=\sqrt 8 = 2\sqrt2 — the street exactly spans the gap. Every other point satisfies yi(w⊤xi+b)>1y_i(\mathbf{w}^\top\mathbf{x}_i+b)>1 strictly, so moving them (while staying on their side) changes nothing.

Python
import numpy as np
from sklearn.svm import SVC

X = np.array([[2,2], [3,3], [4,4], [0,0], [-1,-1], [-2,-2]])
y = np.array([1, 1, 1, -1, -1, -1])

svm = SVC(kernel='linear', C=1000)  # large C = hard margin
svm.fit(X, y)

print(f"Weights: {svm.coef_[0]}")          # ~[0.5, 0.5]
print(f"Bias: {svm.intercept_[0]}")        # ~-1.0
print(f"Margin: {2/np.linalg.norm(svm.coef_[0]):.3f}")   # ~2.83
print(f"Support vectors:\n{svm.support_vectors_}")        # (2,2) and (0,0)

Maximum-margin classifier (SVM)

feature x₁feature x₂
margin0.71
statusseparates

Among all lines that separate the classes, the SVM picks the one with the widest margin to the nearest points — the support vectors (ringed). A wider margin generalizes better.

Support vectors

Only the points closest to the boundary (the support vectors) define the decision boundary. All other points can be moved without changing the result.

This makes SVMs memory-efficient: at prediction time, you only need the support vectors, not the entire training set.

Why support vectors matter

In our example, only (2,2)(2,2) and (0,0)(0,0) are support vectors. Moving (3,3)(3,3) anywhere (as long as it stays on the correct side) doesn't change the boundary. This is why SVMs generalize well — they focus on the hardest examples.

When the data isn't separable

Everything above assumed a hard margin: every point classified correctly, with room to spare. Real datasets have label noise and overlap — demand perfect separation and either no solution exists, or one mislabeled point swings the whole boundary. The fix is the soft margin: let points violate the margin, but charge a price CC per unit of violation. Written without constraints, that price becomes the hinge loss — zero for confidently-correct points, growing linearly for violators.

Slack variables, what CC really tunes, and why a soft-margin SVM is exactly hinge loss + L2 regularization with λ=1/C\lambda = 1/C is the subject of Soft Margins & Tuning C.

SVM vs Logistic Regression

PropertySVMLogistic Regression
Decision boundaryMaximum marginMaximum likelihood
Loss functionHinge lossCross-entropy
Support vectorsOnly boundary pointsAll points
ProbabilitiesNot natural (need calibration)Direct output
Kernel trickYesNo (without explicit features)
Outlier sensitivityModerate (bounded loss)High (unbounded loss)

When to use which

SVMs are great when you have many features and want robust margins. Logistic regression is better when you need calibrated probabilities. For large datasets (> 100K), logistic regression is typically faster.

Exercise

If the weight vector is w = [3, 4], what is the margin of the SVM?

Need a hint?

Margin = 2 / ||w||.

Exercise

Why must you scale features before using an SVM?

Need a hint?

Think about how SVM computes distances.

Common mistakes

  1. Forgetting to scale features: SVMs are sensitive to feature scales. Always standardize.
  2. Not tuning C: The default C=1 may not be optimal. Try [0.01, 0.1, 1, 10, 100].
  3. Using SVM for probability estimates: SVMs output distances, not probabilities. Use SVC(probability=True) with caution (it uses cross-validation internally).

Key takeaways

  • SVM finds the separating hyperplane with the maximum margin to the nearest points.
  • Only the support vectors (points on the margin) define the boundary.
  • Real data isn't separable — the soft margin (next-next lesson) prices violations via CC and the hinge loss.
  • Standardize features first; SVMs live and die by distances.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Maximum Margin ClassificationLinear & Logistic Regress…Linear Algebra for MLLogistic RegressionSoft Margins & Tuning CThe Kernel TrickConstrained Optimization …Loss FunctionsThe ML Project Loop