Support Vector Machines

A support vector machine (SVM) finds the decision boundary that maximizes the margin - the distance between the boundary and the closest points of each class, the "support vectors." Maximizing the margin is the bias term: a wide margin generalizes better than a narrow one fitted tightly to the data. The kernel trick then lets the same margin logic work in curved decision boundaries by implicitly mapping the data into a higher-dimensional space, without ever computing the mapping explicitly. The result is a powerful, well-theorized model for medium-sized datasets - and one whose two hyperparameters (C and gamma) need real attention.

The Margin, Intuitively

Among all the lines that separate two classes, the SVM chooses the one farthest from the nearest points of each class. Those nearest points are the support vectors - the only data that matters to the final boundary, which is why a plot of the support vectors shows the model's "decision-critical" data. Soft-margin SVMs (the C parameter controls this) allow some points to sit inside the margin or on the wrong side, trading margin width for misclassification - the right trade when the classes are not perfectly separable, which is always.

Fitting an SVM Classifier

from sklearn.datasets import make_moons, make_circles
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=500, noise=0.2, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

scaler = StandardScaler().fit(Xtr)
svm = SVC(kernel="rbf", C=1.0, gamma="scale", random_state=0)
svm.fit(scaler.transform(Xtr), ytr)
print("test acc:", round(accuracy_score(yte, svm.predict(scaler.transform(Xte))), 3))
print("support vectors:", svm.n_support_)   # the two closest points per class

Scaling is not optional for SVMs: the margin is a geometric object, and unscaled features distort the geometry exactly as they distort KNN's distances. n_support_ reports how many points per class are support vectors - a useful sanity check (a huge number suggests C is too small or the data is messy).

The Kernels

The RBF kernel is the workhorse: it measures local similarity and can bend the boundary to fit clustered, non-linear data. The linear kernel is the right call on text and other high-dimensional sparse problems, where the data is already effectively in a high-dimensional space and a flat separator is all that is needed.

C and Gamma

from sklearn.model_selection import GridSearchCV

grid = {"C": [0.1, 1, 10, 100], "gamma": ["scale", "auto", 0.01, 0.1]}
search = GridSearchCV(SVC(random_state=0), grid, cv=5, n_jobs=-1)
search.fit(scaler.transform(Xtr), ytr)
print(search.best_params_, round(search.best_score_, 4))

C is the misclassification penalty: small C allows a wide margin with more points inside it (more bias, less variance); large C forces the margin to respect almost every point (less bias, more variance - overfitting on noisy data). gamma (RBF only) sets the reach of each training point: small gamma means a smooth, global boundary; large gamma means each point's influence is local and the boundary becomes wiggly. The two interact - tune them as a pair on a log grid, and expect the sweet spot somewhere in the middle of the range.

Strengths and Limits

• Strengths: strong on medium-sized, clearly structured data; the margin theory is the best-earned in the field; the linear variant is competitive with logistic regression on text.
• Scaling limit: training is roughly cubic in the number of samples - fine at 10k rows, painful at 1M. Beyond that, linear SVMs (SGDClassifier with a hinge loss) or gradient boosting take over.
• No native probabilities: predict_proba uses Platt scaling, a calibration layer that is reasonable but not as principled as a logistic model's.

Summary

Use SVC on scaled, medium-sized data where the classes have a clear (possibly curved) structure: RBF kernel by default, linear kernel for text and high-dimensional sparse features, and C plus gamma tuned together on a log grid with cross-validation. Read n_support_ as a health check, expect the model to slow down past tens of thousands of rows, and remember that on large tabular data, gradient boosting usually does the same job faster - the SVM's territory is the structured medium-sized problem where its margin does real work.