Maximum Margin Classification (SVMs)
Support Vector Machines find the separating hyperplane with the widest possible margin — the "street" between classes. Wider streets generalize better. That "maximize the margin" rule is the loop's objective slot made unusually vivid: it's a precise definition of what a good boundary is, and the whole method follows from it.
Real-world examples: Text classification (spam detection), bioinformatics (protein classification), handwriting recognition, image classification.
The margin
The margin is the distance between the decision boundary and the nearest data points from each class:
Where does come from?
The signed distance from a point to the hyperplane is (project onto the unit normal ). We are free to rescale so that the closest points satisfy — this fixes the otherwise-arbitrary scale. Then:
- the margin hyperplane is ,
- the margin hyperplane is .
The distance from the boundary to either is , so the full street width is . Maximizing the margin therefore means minimizing — equivalently , which is smooth. The SVM solves this minimization subject to every point being correctly classified with room to spare:
Intuition
Imagine you're placing a street between two neighborhoods. The wider the street, the more confident you are that new houses will be on the correct side. SVM finds the widest possible street.
Worked example (solved by hand)
Two classes along the line :
- Class :
- Class :
The closest opposing pair is and — these are the support vectors. By symmetry the boundary is their perpendicular bisector, with normal . Write and impose the margin equations on the two support vectors:
Solving: , then , so , . The boundary is , i.e. .
Margin: , so
which equals the distance between the two support vectors — the street exactly spans the gap. Every other point satisfies strictly, so moving them (while staying on their side) changes nothing.
import numpy as np
from sklearn.svm import SVC
X = np.array([[2,2], [3,3], [4,4], [0,0], [-1,-1], [-2,-2]])
y = np.array([1, 1, 1, -1, -1, -1])
svm = SVC(kernel='linear', C=1000) # large C = hard margin
svm.fit(X, y)
print(f"Weights: {svm.coef_[0]}") # ~[0.5, 0.5]
print(f"Bias: {svm.intercept_[0]}") # ~-1.0
print(f"Margin: {2/np.linalg.norm(svm.coef_[0]):.3f}") # ~2.83
print(f"Support vectors:\n{svm.support_vectors_}") # (2,2) and (0,0)
Maximum-margin classifier (SVM)
Among all lines that separate the classes, the SVM picks the one with the widest margin to the nearest points — the support vectors (ringed). A wider margin generalizes better.
Support vectors
Only the points closest to the boundary (the support vectors) define the decision boundary. All other points can be moved without changing the result.
This makes SVMs memory-efficient: at prediction time, you only need the support vectors, not the entire training set.
Why support vectors matter
In our example, only and are support vectors. Moving anywhere (as long as it stays on the correct side) doesn't change the boundary. This is why SVMs generalize well — they focus on the hardest examples.
When the data isn't separable
Everything above assumed a hard margin: every point classified correctly, with room to spare. Real datasets have label noise and overlap — demand perfect separation and either no solution exists, or one mislabeled point swings the whole boundary. The fix is the soft margin: let points violate the margin, but charge a price per unit of violation. Written without constraints, that price becomes the hinge loss — zero for confidently-correct points, growing linearly for violators.
Slack variables, what really tunes, and why a soft-margin SVM is exactly hinge loss + L2 regularization with is the subject of Soft Margins & Tuning C.
SVM vs Logistic Regression
| Property | SVM | Logistic Regression |
|---|---|---|
| Decision boundary | Maximum margin | Maximum likelihood |
| Loss function | Hinge loss | Cross-entropy |
| Support vectors | Only boundary points | All points |
| Probabilities | Not natural (need calibration) | Direct output |
| Kernel trick | Yes | No (without explicit features) |
| Outlier sensitivity | Moderate (bounded loss) | High (unbounded loss) |
When to use which
SVMs are great when you have many features and want robust margins. Logistic regression is better when you need calibrated probabilities. For large datasets (> 100K), logistic regression is typically faster.
Exercise
If the weight vector is w = [3, 4], what is the margin of the SVM?
Need a hint?
Margin = 2 / ||w||.
Exercise
Why must you scale features before using an SVM?
Need a hint?
Think about how SVM computes distances.
Common mistakes
- Forgetting to scale features: SVMs are sensitive to feature scales. Always standardize.
- Not tuning C: The default C=1 may not be optimal. Try [0.01, 0.1, 1, 10, 100].
- Using SVM for probability estimates: SVMs output distances, not probabilities. Use
SVC(probability=True)with caution (it uses cross-validation internally).
Key takeaways
- SVM finds the separating hyperplane with the maximum margin to the nearest points.
- Only the support vectors (points on the margin) define the boundary.
- Real data isn't separable — the soft margin (next-next lesson) prices violations via and the hinge loss.
- Standardize features first; SVMs live and die by distances.
Related concepts
- The Kernel Trick: going non-linear with kernels
- Soft Margins & Tuning C: what happens when the street must bend
- Logistic Regression: a different objective for a linear boundary
- Constrained Optimization & KKT: where the SVM dual and support-vector sparsity come from