Soft Margins & Tuning C
The maximum-margin classifier assumed the two classes could be perfectly separated. Real datasets have label noise, outliers, and overlap — demand perfect separation and either no solution exists, or a single mislabeled point swings the entire boundary. The soft margin fixes this by making violations possible, but priced.
Slack variables
Give every training point a slack measuring how badly it violates the margin:
- — the point is safely outside the margin (no violation)
- — inside the margin, but still on the correct side
- — fully misclassified
The objective charges for total slack:
C is the price of a violation
sets how much one unit of slack costs relative to margin width:
| Small | Large | |
|---|---|---|
| Violations | cheap: tolerated freely | expensive: avoided at all costs |
| Margin | wide, ignores stragglers | narrow, wraps around every point |
| Bias / variance | higher bias, lower variance | lower bias, higher variance |
| Outlier sensitivity | robust | one outlier can reshape the boundary |
As you recover the hard-margin SVM: infinite price means zero tolerance. As , the margin grows until the data barely matters.
Worked example: computing the penalty
Suppose after training we find (margin width ) and three points have slack values (safe), (inside margin, correct side), (misclassified).
- With : objective =
- With : objective =
At the solver would shrink the margin (increase ) or reclassify point 3 to reduce the violation penalty. At the penalty barely registers, so the wide margin is worth tolerating both violations.
C and γ tune together
With an RBF kernel (see the kernel trick), and interact: large + large is a recipe for memorizing the training set, while small + small underfits to a nearly linear, very wide margin. Grid-search them jointly on a log scale (e.g. , ) with cross-validation.
Feel the dial yourself — every position of the C slider retrains the SVM from scratch on the same overlapping clusters:
Soft margins: C sets the price of a violation
Yellow rings mark margin violators (0 < ξ ≤ 1, inside the band but on the right side); rose rings mark misclassified points (ξ > 1, across the boundary). Small C buys a wide margin by tolerating violations; large C makes each violation expensive, so the margin narrows to exclude them.
The hinge loss view
The same optimization can be written without constraints, as a loss function:
The term is the hinge loss: zero for points beyond the margin, growing linearly for violators. Two things fall out of this view:
- Points outside the margin contribute nothing — gradient zero. Only margin violators and boundary points (the support vectors) shape the solution.
- The SVM is "just" a linear model with hinge loss plus L2 regularization, where plays the role of the regularization strength . The same bias–variance dial as everywhere else.
Plotting hinge against its neighbours makes the shape memorable — a free-pass region beyond the margin, then a straight-line penalty, versus log-loss which never reaches zero:
import numpy as np
import matplotlib.pyplot as plt
z = np.linspace(-2, 3, 100)
hinge = np.maximum(0, 1 - z)
zero_one = (z < 0).astype(float) # 0-1 loss: 1 iff misclassified (y·f(x) < 0)
log_loss = -np.log(np.clip(1/(1+np.exp(-z)), 0.01, 0.99))
plt.plot(z, hinge, label='Hinge loss', linewidth=2)
plt.plot(z, zero_one, label='0-1 loss', linewidth=2, linestyle='--')
plt.plot(z, log_loss, label='Log loss', linewidth=2, linestyle=':')
plt.xlabel('y · f(x)'); plt.ylabel('Loss'); plt.legend()
plt.title('Hinge vs 0-1 vs log loss'); plt.grid(True, alpha=0.3)
plt.show()
Exercise
A training point has slack ξ = 1.4 in a soft-margin SVM. Where is it?
Need a hint?
ξ = 0 is outside the margin; ξ = 1 is exactly on the decision boundary.
Exercise
In a soft margin SVM, what happens when you increase C?
Need a hint?
C controls the tradeoff between margin width and misclassification.
Exercise
Set the SVM soft-margin penalty C for a noisy, overlapping 2-class dataset.
Need a hint?
C=0 ignores all violations. Very large C tries to classify everything correctly (overfits to noise).
Key takeaways
- Slack variables measure margin violations; the soft margin permits them at a price
- is that price: small → wide tolerant margin, large → strict narrow margin, → hard margin
- Soft-margin SVM ≡ hinge loss + L2 regularization with
- Tune (and for RBF) jointly, on a log grid, with cross-validation
Related concepts
- Maximum Margin Classification: the hard-margin ideal this lesson relaxes
- The Kernel Trick: the that must be tuned with
- Loss Functions: hinge alongside MSE, MAE, and cross-entropy