Hyperparameter Tuning

Hyperparameters are the knobs you set before training - the regularization strength, the tree depth, the number of neighbors - and tuning is the search over those knobs guided by validation performance. scikit-learn provides the two standard search engines, GridSearchCV (exhaustive over a specified grid) and RandomizedSearchCV (random draws from distributions), both wrapping cross-validation so the choice is made on honest, repeated estimates rather than a single split. The method is mechanical; the skill is choosing a sensible search space and reading the results without overfitting the search itself.

GridSearchCV

from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

pipe = make_pipeline(StandardScaler(), SVC(random_state=0))

param_grid = {
    "svc__C": [0.1, 1, 10, 100],
    "svc__gamma": ["scale", 0.01, 0.1, 1],
    "svc__kernel": ["rbf", "linear"],
}

search = GridSearchCV(pipe, param_grid,
                      cv=StratifiedKFold(5, shuffle=True, random_state=0),
                      scoring="roc_auc", n_jobs=-1)
search.fit(Xtr, ytr)

print("best params:", search.best_params_)
print("best CV AUC:", round(search.best_score_, 4))
print("test AUC:", round(search.score(Xte, yte), 4))

The grid is the Cartesian product of the listed values - 4 x 4 x 2 = 32 configurations, each scored with 5-fold CV, so 160 model fits. The __ in the parameter names addresses a step inside a pipeline (the svc__ prefix). scoring sets the optimization target, n_jobs=-1 parallelizes across configurations, and the test-set score at the end is the single final check. Total runtime scales as grid-size times folds - estimate it before launching a big search.

RandomizedSearchCV

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, uniform

dist = {
    "svc__C": loguniform(1e-2, 1e2),     # continuous, log-spaced
    "svc__gamma": loguniform(1e-3, 1e1),
    "svc__kernel": ["rbf", "linear"],
}

rnd = RandomizedSearchCV(pipe, dist, n_iter=40,
                         cv=5, scoring="roc_auc",
                         random_state=0, n_jobs=-1)
rnd.fit(Xtr, ytr)
print(rnd.best_params_)

Random search draws n_iter configurations from the distributions instead of covering a grid. It is the better strategy when the space is continuous or large: it spreads the budget across the whole space instead of allocating it uniformly on a grid that may miss the interesting region, and for two or more parameters it usually finds configurations as good as the grid with a fraction of the fits. Use log-uniform distributions for strength-like parameters (C, learning rates, regularization) - their useful range spans orders of magnitude.

Reading the Results Without Fooling Yourself

import pandas as pd

df = pd.DataFrame(search.cv_results_)
cols = ["params", "mean_test_score", "std_test_score", "mean_fit_time"]
top = df.sort_values("rank_test_score").head(8)
print(top[cols].to_string(index=False))

• Look at the standard deviation. If the top two configurations overlap within a standard error, they are statistically the same - pick the simpler one. Chasing the single best cell is overfitting the search.
• Check fit time. A configuration that is marginally better but 10x slower may not be worth it at prediction scale.
• Reserve the test set. The search sees only the CV folds; the held-out test set is scored once, on the final chosen configuration, and that number is the claim.
• Two-stage refinement. A coarse random search to find the region, then a fine grid around the best values, is the standard pattern for expensive searches.

What to Tune, by Model

Most models have one dominant knob (the regularization or capacity parameter) and two or three secondary ones. A practical search covers the dominant knob on a log grid across two or three orders of magnitude, the secondary knobs at a few values, and leaves everything else at default - that small space contains the real gains, and the rest is noise-chasing.

Summary

Tune with GridSearchCV for small discrete spaces and RandomizedSearchCV (log-uniform distributions for strength parameters) for anything larger, always inside cross-validation with a business-matched scorer and n_jobs=-1. Read the results through the standard deviations, not just the means; keep the test set for one final score; and remember that the honest search is the one that stops when the top configurations are statistically indistinguishable - not when the single best cell is found.