Support Vector Machine
Support Vector Machine (SVM) is a very powerful supervised learning algorithm for classification and regression. SVM works by finding the optimal hyperplane that maximizes the margin between two classes.
Basic Concept
Hyperplane
A hyperplane is a line (2D), a plane (3D), or a hyperplane (in higher dimensions) that separates data into two classes.
Support Vectors
Support vectors are the data points closest to the hyperplane. They determine the position and orientation of the hyperplane.
Margin
The margin is the shortest distance between the hyperplane and the nearest data point of each class. The goal of SVM: maximize the margin.
from sklearn.svm import SVC
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
X, y = make_blobs(n_samples=100, centers=2, random_state=42, cluster_std=1.5)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = SVC(kernel='linear', C=1.0)
model.fit(X_train, y_train)
print(f"Accuracy: {model.score(X_test, y_test):.4f}")
print(f"Number of support vectors: {model.n_support_}")
Kernel Trick
When data is not linearly separable, the kernel trick maps the data to a higher dimension where it can be separated linearly.
| Kernel | Function | When to Use |
|---|---|---|
| Linear | K(x,y) = x·y | Linearly separable |
| RBF | K(x,y) = exp(-γ||x-y||²) | Default, non-linear |
| Polynomial | K(x,y) = (γx·y + r)^d | Feature interactions |
| Sigmoid | K(x,y) = tanh(γx·y + r) | Neural network-like |
from sklearn.svm import SVC
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X, y = make_moons(n_samples=200, noise=0.2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
kernels = ['linear', 'rbf', 'poly', 'sigmoid']
for kernel in kernels:
model = SVC(kernel=kernel, random_state=42)
model.fit(X_train, y_train)
acc = model.score(X_test, y_test)
print(f"Kernel {kernel:10s} → Accuracy: {acc:.4f}")
Important Parameters
| Parameter | Explanation | Default |
|---|---|---|
| C | Regularization (smaller → smoother) | 1.0 |
| gamma | Kernel influence (larger → more complex) | 'scale' |
| kernel | Kernel type | 'rbf' |
from sklearn.svm import SVC
# Small C: wide margin, better generalization
model_low_c = SVC(C=0.01, kernel='rbf')
# Large C: narrow margin, fits training data more tightly
model_high_c = SVC(C=100, kernel='rbf')
# Small gamma: far-reaching influence
model_low_g = SVC(gamma=0.01, kernel='rbf')
# Large gamma: local influence only
model_high_g = SVC(gamma=100, kernel='rbf')
Soft Margin vs Hard Margin
| Type | Explanation |
|---|---|
| Hard Margin | Zero tolerance for misclassification |
| Soft Margin | Allows some errors (controlled by C) |
Multi-Class with SVM
SVM is naturally a binary classifier. For multi-class problems, sklearn uses the One-vs-One or One-vs-Rest strategy:
from sklearn.datasets import load_iris
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
iris = load_iris()
X, y = iris.data, iris.target
pipeline = make_pipeline(StandardScaler(), SVC(kernel='rbf', C=1.0))
pipeline.fit(X, y)
print(f"Accuracy: {pipeline.score(X, y):.4f}")
print(f"Shape support vectors: {pipeline[1].support_vectors_.shape}")
Pros and Cons
| Pros | Cons |
|---|---|
| Effective in high dimensions | Slow on large datasets |
| Memory efficient (only support vectors) | Hard to interpret |
| Versatile (multiple kernels) | Sensitive to scaling |
| Overfitting minimized with regularization | Parameter selection is crucial |
SVM is very powerful for classification. In the next chapter we will cover Decision Tree, a more interpretable decision-tree-based algorithm.