Hyperparameter Tuning

Hyperparameter Tuning is the process of finding the best combination of hyperparameters for a model. Unlike parameters that are learned from the data, hyperparameters are set before training.

Parameters vs Hyperparameters

Grid Search

Grid Search evaluates every combination of hyperparameters exhaustively.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [3, 5, 10, None],
    'min_samples_split': [2, 5, 10]
}

grid = GridSearchCV(RandomForestClassifier(random_state=42),
                    param_grid, cv=5, scoring='accuracy',
                    n_jobs=-1, verbose=1)
grid.fit(X, y)

print(f"Best params: {grid.best_params_}")
print(f"Best score: {grid.best_score_:.4f}")

Random Search

Random Search evaluates random combinations drawn from hyperparameter distributions. It is more efficient than Grid Search for large parameter spaces.

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from scipy.stats import randint, uniform

iris = load_iris()
X, y = iris.data, iris.target

param_dist = {
    'n_estimators': randint(50, 300),
    'max_depth': randint(3, 20),
    'min_samples_split': randint(2, 20),
    'min_samples_leaf': randint(1, 10),
    'max_features': uniform(0.1, 0.9)
}

random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_dist, n_iter=50, cv=5,
    scoring='accuracy', random_state=42, n_jobs=-1
)
random_search.fit(X, y)

print(f"Best params: {random_search.best_params_}")
print(f"Best score: {random_search.best_score_:.4f}")

Comparing Grid vs Random Search

Bayesian Optimization (Optuna)

import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

def objective(trial):
    n_estimators = trial.suggest_int('n_estimators', 50, 300)
    max_depth = trial.suggest_int('max_depth', 3, 20)
    min_samples_split = trial.suggest_int('min_samples_split', 2, 20)

    model = RandomForestClassifier(
        n_estimators=n_estimators,
        max_depth=max_depth,
        min_samples_split=min_samples_split,
        random_state=42
    )

    scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
    return scores.mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50)

print(f"Best trial: {study.best_trial.params}")
print(f"Best score: {study.best_value:.4f}")

Hyperopt

from hyperopt import fmin, tpe, hp, STATUS_OK
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

space = {
    'n_estimators': hp.quniform('n_estimators', 50, 300, 10),
    'max_depth': hp.quniform('max_depth', 3, 20, 1),
    'min_samples_split': hp.quniform('min_samples_split', 2, 20, 1)
}

def objective(params):
    model = RandomForestClassifier(
        n_estimators=int(params['n_estimators']),
        max_depth=int(params['max_depth']),
        min_samples_split=int(params['min_samples_split']),
        random_state=42
    )
    scores = cross_val_score(model, X, y, cv=5)
    return {'loss': -scores.mean(), 'status': STATUS_OK}

best = fmin(objective, space, algo=tpe.suggest, max_evals=50)
print(f"Best: {best}")

Best Practices

• Start with Random Search for initial exploration
• Use Grid Search in a narrow range for fine-tuning
• Use Bayesian optimization (Optuna) for high efficiency
• Always use cross validation, not a single split
• Use n_iter of at least 30-50 for Random Search
• Log every experiment for reproducibility

Method Summary

Next chapter: Overfitting & Underfitting, the most common problems in ML.

Next: Overfitting & Underfitting →