Hyperparameter Tuning

Logistic regression has few hyperparameters, which is a feature: the search space is small enough to tune exhaustively in minutes. The two dials are the penalty strength (C) and the penalty type (l1/l2/elasticnet); the solver is a compatibility choice, not a quality dial. The tool for all of it is cross-validated grid search, which picks the configuration by validated performance instead of by feel.

GridSearchCV

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=12, n_informative=6,
                           random_state=6)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression(max_iter=3000)),
])

grid = {
    "model__penalty": ["l1", "l2"],
    "model__C": [0.01, 0.1, 1.0, 10.0, 100.0],
    "model__solver": ["liblinear"],   # handles both penalties
}

search = GridSearchCV(pipe, grid, cv=StratifiedKFold(5),
                      scoring="roc_auc", n_jobs=-1)
search.fit(Xtr, ytr)

print("best params:", search.best_params_)
print("test AUC:", search.score(Xte, yte))

The pipeline keeps the scaler honest: each CV fold fits its own scaler on its own training portion. The stratified folds preserve the class ratio, and scoring="roc_auc" optimizes ranking quality instead of accuracy at one fixed threshold.

Reading the Results

import pandas as pd

df = pd.DataFrame(search.cv_results_)
cols = ["params", "mean_test_score", "std_test_score"]
print(df[cols].to_string(index=False))

Look at the standard deviations, not just the means. If the top two configurations overlap within one standard error, they are the same model statistically - prefer the simpler one (stronger regularization, fewer features active). Chasing the single best cell in a noisy table is overfitting the search itself.

RandomizedSearchCV

When the grid grows (many C values times multiple solvers), a random draw from distributions explores the space more efficiently per evaluation:

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, choice

dist = {
    "model__C": loguniform(1e-2, 1e2),
    "model__penalty": choice(["l1", "l2"]),
}
rnd = RandomizedSearchCV(pipe, dist, n_iter=30, cv=5,
                         scoring="roc_auc", random_state=0, n_jobs=-1)
rnd.fit(Xtr, ytr)
print(rnd.best_params_)

With only two meaningful parameters, the full grid is cheap and preferred; random search earns its keep when you add elasticnet's l1_ratio or start comparing model families.

The Full Checklist

• C on a log scale (1e-2 to 1e2 covers virtually everything).
• penalty: l2 vs l1 - l1 often wins when features are mostly noise.
• class_weight: None vs balanced, on imbalanced data.
• Solver last, and only for compatibility (l1 needs liblinear/saga/sag).
• Never tune on the test set - the search uses CV folds; the test set sees the final model exactly once.

Summary

Tune C on a log grid with StratifiedKFold and an AUC or log-loss scorer inside a Pipeline that carries the scaler. Trust configurations only if they beat the field by more than their noise, keep the test set untouched until the final fit, and stop early - with two real dials, the optimal logistic regression is usually within a few grid cells of the default.