🎯 What You'll Learn
- Understand hyperplanes, margins, and why maximizing the margin leads to better generalization
- Identify support vectors and explain why only they define the decision boundary
- Distinguish hard-margin vs soft-margin SVMs and understand the C regularization parameter
- Apply the kernel trick to handle non-linearly separable data without explicit feature mapping
- Tune C and γ for the RBF kernel, and understand why feature scaling is mandatory for SVMs
1 The Hyperplane
A hyperplane is the generalization of a line or plane to N dimensions. In 2D, it's a line. In 3D, it's a flat surface. In N dimensions, it's an (N-1)-dimensional surface. For classification, the hyperplane serves as the decision boundary: data points on one side belong to class +1, and data points on the other side belong to class −1.
Many possible hyperplanes can separate two linearly separable classes. A naive approach might draw any line that separates them. The question SVM asks is: which hyperplane will generalize best to unseen data?
The answer is the hyperplane that maximises the margin — the distance from the hyperplane to the nearest data points of each class. Intuitively, a wider margin means the model has more "breathing room" between classes, making it less sensitive to small perturbations in the data. This is the key insight that gives SVMs their strong theoretical guarantees.
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report
import numpy as np
import matplotlib.pyplot as plt
# Generate linearly separable 2D dataset
X, y = make_classification(n_samples=100, n_features=2, n_redundant=0,
n_informative=2, n_clusters_per_class=1,
class_sep=2.0, random_state=42)
y_signed = 2 * y - 1 # Convert 0/1 labels to -1/+1 for SVM convention
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)
# Train a linear SVM
svm_linear = SVC(kernel='linear', C=1.0)
svm_linear.fit(X_train_s, y_train)
print("Linear SVM accuracy:", svm_linear.score(X_test_s, y_test))
print(f"Number of support vectors per class: {svm_linear.n_support_}")
print(f"Total support vectors: {len(svm_linear.support_vectors_)}")
print(f"Hyperplane coefficients (w): {svm_linear.coef_[0].round(4)}")
print(f"Hyperplane bias (b): {svm_linear.intercept_[0]:.4f}")2 Maximum Margin Classifier
The SVM finds the hyperplane that maximises the geometric margin — the perpendicular distance from the hyperplane to the nearest points of each class. The hyperplane is positioned equidistant from both classes' nearest points.
Mathematically, if the hyperplane is defined by w · x + b = 0, then the margin is 2 / ‖w‖. Maximizing the margin is equivalent to minimizing ‖w‖². This becomes a quadratic programming (QP) problem with linear constraints.
The elegance of this formulation: the optimal solution depends only on a small subset of training points — the support vectors. The rest of the data is irrelevant to the final model. This is computationally and theoretically important.
Maximizing margin is not just a mathematical convenience — it has deep theoretical justification in statistical learning theory (VC theory). A larger margin corresponds to a lower VC dimension, which bounds the generalization error. This is why SVMs often outperform logistic regression on small datasets: the maximum-margin criterion is a stronger regularization principle than L2 alone.
The solid line is the maximum-margin hyperplane. The two dashed lines are the margin boundaries, each at distance 1/‖w‖ from the hyperplane. Only the highlighted support vectors (touching the dashed lines) determine where the hyperplane sits — every other point could move or vanish without changing the solution.
3 Support Vectors
The support vectors are the training data points that lie exactly on the margin boundaries — the closest points to the hyperplane. They "support" the hyperplane in the sense that removing any non-support-vector from the training set would not change the model at all. The hyperplane is completely defined by the support vectors alone.
This property has important practical consequences:
- SVMs are robust to outliers that are far from the decision boundary
- The model's memory footprint at inference time depends only on the number of support vectors, not the full training set
- When you retrain on new data, only new points near the boundary affect the model
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
import numpy as np
X, y = make_classification(n_samples=200, n_features=2, n_redundant=0,
n_informative=2, n_clusters_per_class=1,
class_sep=1.5, random_state=42)
scaler = StandardScaler()
X_s = scaler.fit_transform(X)
svm = SVC(kernel='linear', C=1.0)
svm.fit(X_s, y)
print(f"Total training samples: {len(X_s)}")
print(f"Number of support vectors: {len(svm.support_vectors_)} "
f"({100*len(svm.support_vectors_)/len(X_s):.1f}% of training data)")
# Verify: support vectors are the points closest to the hyperplane
# Distance of each training point to the hyperplane (w·x + b)
w = svm.coef_[0]
b = svm.intercept_[0]
dists = np.abs(X_s @ w + b) / np.linalg.norm(w)
# Support vectors have the minimum distance (distance = 1/||w|| from the hyperplane)
sv_dists = dists[svm.support_]
non_sv_dists = np.delete(dists, svm.support_)
print(f"\nMean distance to boundary — support vectors: {sv_dists.mean():.4f}")
print(f"Mean distance to boundary — non-support vectors: {non_sv_dists.mean():.4f}")
print(f"Max distance of support vectors: {sv_dists.max():.4f}")
print(f"Min distance of non-support vectors: {non_sv_dists.min():.4f}")4 Hard Margin vs Soft Margin (C Parameter)
The original SVM formulation requires perfect linear separation (hard margin). In practice, real data is rarely perfectly separable, and even when it is, allowing some misclassifications often leads to a wider margin and better generalization. Enter the soft-margin SVM.
Soft-margin SVM introduces slack variables (ξᵢ) for each training point, allowing some points to violate the margin or even be misclassified. The objective becomes:
Minimize: ½‖w‖² + C × Σᵢ ξᵢ
The parameter C controls the tradeoff:
- Large C: You care a lot about misclassifications. The model tries hard to classify all training points correctly — small margin, potential overfitting.
- Small C: You tolerate more training errors to get a wider margin — better generalization, potential underfitting.
- C → ∞: Hard-margin SVM (no violations allowed).
The chart below shows a real linear SVM (sklearn.svm.SVC(kernel='linear')) fitted at four different values of C on the same 24-point 2D dataset, which includes a couple of points that sit awkwardly close to the other class. Click a C value to see how the margin width, the decision boundary, and the set of support vectors all change together:
C = 0.5 — a moderately wide margin tolerating two margin violations.
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np
X, y = make_classification(n_samples=300, n_features=5, noise=0.1,
random_state=42)
C_values = [0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0]
print(f"{'C':>8} {'CV Accuracy':>12} {'Support Vectors':>16}")
print("-" * 42)
for C in C_values:
pipe = Pipeline([
('scaler', StandardScaler()),
('svm', SVC(kernel='linear', C=C))
])
scores = cross_val_score(pipe, X, y, cv=5, scoring='accuracy')
# Fit to get number of SVs
pipe.fit(X, y)
n_sv = len(pipe.named_steps['svm'].support_vectors_)
print(f"{C:>8} {scores.mean():>12.4f} {n_sv:>16}")5 The Kernel Trick
Linear SVMs only work when classes are linearly separable — or nearly so. What if the decision boundary is curved? The classic solution is to map data to a higher-dimensional space where a linear separator exists, then find the hyperplane there. A non-linear boundary in the original space corresponds to a linear boundary in the transformed space.
The computational problem: explicitly computing high-dimensional transformations is extremely expensive. The kernel trick avoids this entirely. SVM optimization only ever needs inner products between data points (xᵢ · xⱼ). If we replace this with a kernel function K(xᵢ, xⱼ) that computes the inner product in the high-dimensional space without ever going there, we get the full power of the high-dimensional mapping at a fraction of the cost.
Common kernels:
- Linear: K(x, z) = x · z. No transformation — same as standard SVM.
- Polynomial: K(x, z) = (γ·x·z + r)^d. Captures polynomial relationships.
- RBF (Radial Basis Function / Gaussian): K(x, z) = exp(−γ‖x−z‖²). Maps to infinite-dimensional space. The most widely used kernel.
- Sigmoid: K(x, z) = tanh(γ·x·z + r). An S-shaped squashing function, cousin of the sigmoid from Lesson 15. Rarely the best choice in practice.
The classic example of why a kernel matters: a "bullseye" dataset, where one class forms a ring around the other. No straight line can separate them, but the RBF kernel finds a curved boundary effortlessly. Toggle between the two real, fitted SVC models below (same C=1.0 for both — only the kernel changes):
Linear kernel — a straight decision boundary cannot separate a ring from its center; accuracy 58.6%.
from sklearn.svm import SVC
from sklearn.datasets import make_circles, make_moons
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import numpy as np
# Circular data: not linearly separable in 2D
X_circles, y_circles = make_circles(n_samples=300, factor=0.5, noise=0.1, random_state=42)
X_moons, y_moons = make_moons(n_samples=300, noise=0.15, random_state=42)
for name, X, y in [("Circles", X_circles, y_circles),
("Moons", X_moons, y_moons)]:
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42)
scaler = StandardScaler()
X_tr_s = scaler.fit_transform(X_tr)
X_te_s = scaler.transform(X_te)
# Linear SVM — will struggle with non-linear data
svm_linear = SVC(kernel='linear', C=1.0)
svm_linear.fit(X_tr_s, y_tr)
# RBF kernel SVM — maps to infinite-dimensional space
svm_rbf = SVC(kernel='rbf', C=1.0, gamma='scale')
svm_rbf.fit(X_tr_s, y_tr)
print(f"\n{name} dataset:")
print(f" Linear SVM accuracy: {svm_linear.score(X_te_s, y_te):.4f}")
print(f" RBF SVM accuracy: {svm_rbf.score(X_te_s, y_te):.4f}")The RBF kernel K(x, z) = exp(−γ‖x−z‖²) implicitly computes a dot product in an infinite-dimensional feature space — the space of all polynomial features of any degree. The SVM never computes the actual transformed features; it only evaluates K(xᵢ, xⱼ) between pairs of training points. This makes the algorithm computationally feasible for arbitrarily complex feature spaces.
6 SVC in scikit-learn
sklearn provides SVC (Support Vector Classifier) for classification and SVR (Support Vector Regressor) for regression. Key parameters:
from sklearn.svm import SVC, SVR
from sklearn.datasets import load_breast_cancer, load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, mean_squared_error
from sklearn.pipeline import Pipeline
import numpy as np
# ---- Classification: SVC ----
cancer = load_breast_cancer()
X, y = cancer.data, cancer.target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
random_state=42, stratify=y)
svc_pipe = Pipeline([
('scaler', StandardScaler()),
('svc', SVC(kernel='rbf', C=10.0, gamma='scale',
probability=True, # enables predict_proba() — slightly slower
random_state=42))
])
svc_pipe.fit(X_tr, y_tr)
print("Classification Report (SVC, RBF kernel):")
print(classification_report(y_te, svc_pipe.predict(X_te),
target_names=cancer.target_names))
# Probability outputs (requires probability=True at construction)
proba = svc_pipe.predict_proba(X_te)[:3]
print("Probability outputs for first 3 test samples:")
print(proba.round(3))
# ---- Regression: SVR ----
diabetes = load_diabetes()
X_d, y_d = diabetes.data, diabetes.target
X_tr_d, X_te_d, y_tr_d, y_te_d = train_test_split(
X_d, y_d, test_size=0.2, random_state=42)
svr_pipe = Pipeline([
('scaler', StandardScaler()),
('svr', SVR(kernel='rbf', C=100.0, gamma='scale', epsilon=0.1))
])
svr_pipe.fit(X_tr_d, y_tr_d)
y_pred_d = svr_pipe.predict(X_te_d)
rmse = np.sqrt(mean_squared_error(y_te_d, y_pred_d))
print(f"\nSVR Regression RMSE: {rmse:.2f}")7 Tuning C and γ with GridSearchCV
The two most critical hyperparameters for an RBF SVM are:
- C: Soft-margin regularization. Small C = wider margin, more violations tolerated. Large C = narrow margin, fewer violations.
- γ (gamma): RBF kernel bandwidth. Large γ: each training point has a narrow "sphere of influence" — tight, wiggly boundary — overfitting. Small γ: each point influences a wide region — smooth, broad boundary — underfitting.
gamma='scale'(default) sets γ = 1/(n_features × X.var()).
C and γ interact: a large C with a large γ will almost always overfit. They should be tuned together — and trying every combination by hand would mean writing a nested loop of the cross-validation searches you've been doing since Lesson 13.
GridSearchCV — the search loop, automated
You already know the manual pattern: loop over candidate settings, run cross_val_score for each, keep the winner. GridSearchCV does exactly that for you: give it a model and a grid of parameter values, and it cross-validates every combination (here 4 × 4 = 16 combos × 5 folds = 80 model fits), then exposes the winner as .best_params_. That's the whole trick — nothing new conceptually. Lesson 27 is devoted to using it (and its faster cousins) well.
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
import numpy as np
cancer = load_breast_cancer()
X, y = cancer.data, cancer.target
# Pipeline to prevent data leakage during CV
pipe = Pipeline([
('scaler', StandardScaler()),
('svc', SVC(kernel='rbf', random_state=42))
])
# Grid of hyperparameters
param_grid = {
'svc__C': [0.1, 1.0, 10.0, 100.0],
'svc__gamma': ['scale', 0.001, 0.01, 0.1]
}
grid_search = GridSearchCV(pipe, param_grid, cv=5, scoring='accuracy',
n_jobs=-1, verbose=1)
grid_search.fit(X, y)
print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV accuracy: {grid_search.best_score_:.4f}")
# Results for all combinations
results = grid_search.cv_results_
for C, gamma, mean_score in zip(
results['param_svc__C'],
results['param_svc__gamma'],
results['mean_test_score']):
print(f" C={C:6}, gamma={str(gamma):8} → {mean_score:.4f}")When tuning SVMs with cross-validation, always wrap the scaler and SVM in a Pipeline. If you scale the entire dataset before passing it to GridSearchCV, the test fold's statistics leak into the scaler's fit — contaminating the validation and giving overly optimistic accuracy estimates.
8 Why Feature Scaling is Mandatory
Just like KNN, SVMs rely on distances between data points (via the kernel function). Unscaled features distort these distances and therefore distort the kernel values and the margin calculation. A feature with values in the range [0, 100,000] will completely dominate the RBF kernel's distance calculation, making all other features effectively invisible.
from sklearn.svm import SVC
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np
cancer = load_breast_cancer()
X, y = cancer.data, cancer.target
# Print feature ranges to appreciate the scale differences
for i, name in enumerate(cancer.feature_names[:5]):
print(f"{name:40s} range: [{X[:, i].min():.3f}, {X[:, i].max():.3f}]")
# Without scaling — SVM gets confused by different feature scales
svm_raw = SVC(kernel='rbf', C=1.0, gamma='scale')
scores_raw = cross_val_score(svm_raw, X, y, cv=5, scoring='accuracy')
# With scaling — features on same scale
svm_pipe = Pipeline([
('scaler', StandardScaler()),
('svm', SVC(kernel='rbf', C=1.0, gamma='scale'))
])
scores_scaled = cross_val_score(svm_pipe, X, y, cv=5, scoring='accuracy')
print(f"\nWithout scaling: {scores_raw.mean():.4f} ± {scores_raw.std():.4f}")
print(f"With scaling: {scores_scaled.mean():.4f} ± {scores_scaled.std():.4f}")Real-World Spotlight: Image Classification Before Deep Learning
Before the deep learning revolution (starting with AlexNet in 2012), SVMs with RBF kernels were the dominant approach for image classification. On MNIST (70,000 handwritten digit images), an RBF SVM with tuned C and γ achieved >99% accuracy — remarkable for a relatively simple algorithm.
The approach: flatten each 28×28 image to a 784-dimensional vector, standardize, then train an SVC with OvR multi-class strategy. The 10-class problem trains 10 binary SVMs. GridSearchCV tunes C and γ. Even today, on small datasets, SVMs often outperform neural networks that need more data to overcome their inductive bias.
from sklearn.svm import SVC
from sklearn.datasets import load_digits # 8x8 version of MNIST
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report
import numpy as np
# load_digits: 1797 samples, 64 features (8x8 pixels), 10 classes (0–9)
digits = load_digits()
X, y = digits.data, digits.target # already flattened to 64 features
print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features, {len(np.unique(y))} classes")
print(f"Feature range: [{X.min():.1f}, {X.max():.1f}]")
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
# SVM pipeline
svm_pipe = Pipeline([
('scaler', StandardScaler()),
('svc', SVC(kernel='rbf', C=10.0, gamma=0.001, decision_function_shape='ovr'))
])
svm_pipe.fit(X_train, y_train)
print("\nClassification Report (10-class digit recognition):")
print(classification_report(y_test, svm_pipe.predict(X_test)))
# Compare with softmax regression (LogisticRegression)
from sklearn.linear_model import LogisticRegression
lr_pipe = Pipeline([
('scaler', StandardScaler()),
('lr', LogisticRegression(max_iter=5000, C=1.0, multi_class='multinomial'))
])
lr_scores = cross_val_score(lr_pipe, X, y, cv=5, scoring='accuracy')
svm_scores = cross_val_score(svm_pipe, X, y, cv=5, scoring='accuracy')
print(f"\nSoftmax Regression CV accuracy: {lr_scores.mean():.4f}")
print(f"RBF SVM CV accuracy: {svm_scores.mean():.4f}")Quick Check
✍️ Practice Exercises
- Use
make_circles(noise=0.1)(200 samples). Train a linear SVC and an RBF SVC, both with proper scaling. Report accuracy. Explain why linear fails on this data. - On the breast cancer dataset, perform a GridSearchCV over C=[0.1, 1, 10, 100] and gamma=['scale', 0.001, 0.01] for an RBF SVM. Print the best combination and its 5-fold accuracy.
- Train an SVC with
probability=Trueon the iris dataset. Print the probability outputs for 5 test samples. Then compare these probabilities with those from LogisticRegression — which model's probabilities are better calibrated (closer to 0.5 for uncertain predictions)? - Using the digits dataset, plot a heatmap or table of test accuracy for a grid of C and gamma values. Identify the region of best performance and the region of clear overfitting.
- Demonstrate the mandatory nature of scaling: train an SVR on the diabetes dataset with and without StandardScaler. Report the R² score for both. Explain the performance gap.
▶ GridSearchCV Heatmap Hint
import numpy as np
import matplotlib.pyplot as plt
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_digits
digits = load_digits()
X, y = digits.data, digits.target
Cs = [0.01, 0.1, 1, 10, 100]
gammas = [0.0001, 0.001, 0.01, 0.1]
pipe = Pipeline([('s', StandardScaler()), ('svc', SVC(kernel='rbf'))])
gs = GridSearchCV(pipe, {'svc__C': Cs, 'svc__gamma': gammas}, cv=3)
gs.fit(X, y)
scores = gs.cv_results_['mean_test_score'].reshape(len(Cs), len(gammas))
plt.figure(figsize=(7, 5))
plt.imshow(scores, cmap='viridis', aspect='auto')
plt.colorbar(label='CV Accuracy')
plt.xticks(range(len(gammas)), gammas)
plt.yticks(range(len(Cs)), Cs)
plt.xlabel('gamma'); plt.ylabel('C')
plt.title('SVM Hyperparameter Heatmap')
plt.show()📚 Primary Sources
sklearn: Support Vector Machines — excellent documentation with kernel math and practical guidance.
Andrew Ng's CS229 SVM Notes — the most thorough and accessible mathematical derivation available.