Soft Margins & Tuning COpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Soft Margins & Tuning C

The maximum-margin classifier assumed the two classes could be perfectly separated. Real datasets have label noise, outliers, and overlap — demand perfect separation and either no solution exists, or a single mislabeled point swings the entire boundary. The soft margin fixes this by making violations possible, but priced.

Slack variables

Give every training point a slack ξi≥0\xi_i \geq 0 measuring how badly it violates the margin:

yi(w⊤xi+b)≥1−ξiy_i(\mathbf{w}^\top \mathbf{x}_i + b) \geq 1 - \xi_i
  • ξi=0\xi_i = 0 — the point is safely outside the margin (no violation)
  • 0<ξi<10 < \xi_i < 1 — inside the margin, but still on the correct side
  • ξi>1\xi_i > 1 — fully misclassified

The objective charges for total slack:

min⁡w, b, ξ    12∥w∥2+C∑i=1nξi\min_{\mathbf{w},\, b,\, \boldsymbol{\xi}} \;\; \frac{1}{2}\|\mathbf{w}\|^2 + C \sum_{i=1}^{n} \xi_i

C is the price of a violation

CC sets how much one unit of slack costs relative to margin width:

Small CCLarge CC
Violationscheap: tolerated freelyexpensive: avoided at all costs
Marginwide, ignores stragglersnarrow, wraps around every point
Bias / variancehigher bias, lower variancelower bias, higher variance
Outlier sensitivityrobustone outlier can reshape the boundary

As C→∞C \to \infty you recover the hard-margin SVM: infinite price means zero tolerance. As C→0C \to 0, the margin grows until the data barely matters.

Worked example: computing the penalty

Suppose after training we find ∥w∥2=4\|\mathbf{w}\|^2 = 4 (margin width 2/∥w∥=12/\|\mathbf{w}\| = 1) and three points have slack values ξ1=0\xi_1 = 0 (safe), ξ2=0.6\xi_2 = 0.6 (inside margin, correct side), ξ3=1.4\xi_3 = 1.4 (misclassified).

  • With C=0.1C = 0.1: objective = 12(4)+0.1(0+0.6+1.4)=2+0.2=2.2\tfrac{1}{2}(4) + 0.1(0 + 0.6 + 1.4) = 2 + 0.2 = 2.2
  • With C=10C = 10: objective = 12(4)+10(0+0.6+1.4)=2+20=22\tfrac{1}{2}(4) + 10(0 + 0.6 + 1.4) = 2 + 20 = 22

At C=10C=10 the solver would shrink the margin (increase ∥w∥2\|\mathbf{w}\|^2) or reclassify point 3 to reduce the violation penalty. At C=0.1C=0.1 the penalty barely registers, so the wide margin is worth tolerating both violations.

C and γ tune together

With an RBF kernel (see the kernel trick), CC and γ\gamma interact: large γ\gamma + large CC is a recipe for memorizing the training set, while small γ\gamma + small CC underfits to a nearly linear, very wide margin. Grid-search them jointly on a log scale (e.g. C∈10−2…103C \in 10^{-2} \ldots 10^{3}, γ∈10−4…101\gamma \in 10^{-4} \ldots 10^{1}) with cross-validation.

Feel the dial yourself — every position of the C slider retrains the SVM from scratch on the same overlapping clusters:

Soft margins: C sets the price of a violation

feature x₁feature x₂
● class +1● class −1○ violator (0 < ξ ≤ 1)○ misclassified (ξ > 1)
C1.00
margin width 2/‖w‖1.98
violators5
misclassified3
Σξ7.14

Yellow rings mark margin violators (0 < ξ ≤ 1, inside the band but on the right side); rose rings mark misclassified points (ξ > 1, across the boundary). Small C buys a wide margin by tolerating violations; large C makes each violation expensive, so the margin narrows to exclude them.

The hinge loss view

The same optimization can be written without constraints, as a loss function:

L=12∥w∥2+C∑imax⁡ ⁣(0,  1−yi(w⊤xi+b))\mathcal{L} = \frac{1}{2}\|\mathbf{w}\|^2 + C \sum_i \max\!\big(0,\; 1 - y_i(\mathbf{w}^\top \mathbf{x}_i + b)\big)

The max⁡(0,1−margin)\max(0, 1 - \text{margin}) term is the hinge loss: zero for points beyond the margin, growing linearly for violators. Two things fall out of this view:

  1. Points outside the margin contribute nothing — gradient zero. Only margin violators and boundary points (the support vectors) shape the solution.
  2. The SVM is "just" a linear model with hinge loss plus L2 regularization, where 1/C1/C plays the role of the regularization strength λ\lambda. The same bias–variance dial as everywhere else.

Plotting hinge against its neighbours makes the shape memorable — a free-pass region beyond the margin, then a straight-line penalty, versus log-loss which never reaches zero:

Python
import numpy as np
import matplotlib.pyplot as plt

z = np.linspace(-2, 3, 100)
hinge = np.maximum(0, 1 - z)
zero_one = (z < 0).astype(float)          # 0-1 loss: 1 iff misclassified (y·f(x) < 0)
log_loss = -np.log(np.clip(1/(1+np.exp(-z)), 0.01, 0.99))

plt.plot(z, hinge, label='Hinge loss', linewidth=2)
plt.plot(z, zero_one, label='0-1 loss', linewidth=2, linestyle='--')
plt.plot(z, log_loss, label='Log loss', linewidth=2, linestyle=':')
plt.xlabel('y · f(x)'); plt.ylabel('Loss'); plt.legend()
plt.title('Hinge vs 0-1 vs log loss'); plt.grid(True, alpha=0.3)
plt.show()

Exercise

A training point has slack ξ = 1.4 in a soft-margin SVM. Where is it?

Need a hint?

ξ = 0 is outside the margin; ξ = 1 is exactly on the decision boundary.

Exercise

In a soft margin SVM, what happens when you increase C?

Need a hint?

C controls the tradeoff between margin width and misclassification.

Exercise

Set the SVM soft-margin penalty C for a noisy, overlapping 2-class dataset.

0.0150.01100
Need a hint?

C=0 ignores all violations. Very large C tries to classify everything correctly (overfits to noise).

Key takeaways

  • Slack variables ξi\xi_i measure margin violations; the soft margin permits them at a price
  • CC is that price: small CC → wide tolerant margin, large CC → strict narrow margin, C→∞C \to \infty → hard margin
  • Soft-margin SVM ≡ hinge loss + L2 regularization with λ=1/C\lambda = 1/C
  • Tune CC (and γ\gamma for RBF) jointly, on a log grid, with cross-validation

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Soft Margins & Tuning CLinear & Logistic Regress…Linear Algebra for MLMaximum Margin Classifica…The Kernel TrickLoss Functions