Cross Validation

Cross Validation is a technique for evaluating a model more robustly by splitting the data into several folds. It reduces the risk of overfitting in evaluation and provides a more accurate performance estimate.

Why Cross Validation?

K-Fold Cross Validation

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

model = RandomForestClassifier(n_estimators=100, random_state=42)
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')

print(f"Scores: {scores}")
print(f"Mean: {scores.mean():.4f}")
print(f"Std: {scores.std():.4f}")

Stratified K-Fold

Ensures the class proportion stays the same in every fold (important for imbalanced datasets).

from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

model = RandomForestClassifier(n_estimators=100, random_state=42)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X, y, cv=skf, scoring='accuracy')
print(f"Stratified K-Fold: {scores.mean():.4f} (+/- {scores.std():.4f})")

Leave-One-Out (LOO)

Each sample becomes the test set once. Suitable for small datasets.

from sklearn.model_selection import LeaveOneOut, cross_val_score
from sklearn.svm import SVC
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

loo = LeaveOneOut()
model = SVC(kernel='rbf')
scores = cross_val_score(model, X, y, cv=loo)

print(f"LOO Accuracy: {scores.mean():.4f}")
print(f"Number of iterations: {len(scores)}")

Time Series Split

For time series data, use TimeSeriesSplit to preserve the temporal order.

from sklearn.model_selection import TimeSeriesSplit, cross_val_score
from sklearn.linear_model import LinearRegression
import numpy as np

np.random.seed(42)
X = np.arange(100).reshape(-1, 1)
y = np.random.randn(100).cumsum()

model = LinearRegression()
tscv = TimeSeriesSplit(n_splits=5)

scores = cross_val_score(model, X, y, cv=tscv)
print(f"Time Series CV Scores: {scores}")
print(f"Mean: {scores.mean():.4f}")

Cross Validation with a Pipeline

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svm', SVC(kernel='rbf'))
])

scores = cross_val_score(pipeline, X, y, cv=5, scoring='accuracy')
print(f"Pipeline CV: {scores.mean():.4f} (+/- {scores.std():.4f})")

Cross Validate (Multi-Metric)

from sklearn.model_selection import cross_validate
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

model = RandomForestClassifier(n_estimators=100, random_state=42)
scoring = ['accuracy', 'precision_macro', 'recall_macro', 'f1_macro']

results = cross_validate(model, X, y, cv=5, scoring=scoring)

for metric in scoring:
    key = f'test_{metric}'
    print(f"{metric:20s}: {results[key].mean():.4f}")

Visualizing the Folds

from sklearn.model_selection import KFold
import numpy as np

X = np.arange(20).reshape(-1, 1)
kf = KFold(n_splits=5, shuffle=True, random_state=42)

for fold, (train_idx, test_idx) in enumerate(kf.split(X)):
    print(f"Fold {fold+1}: Train={train_idx.tolist()}, Test={test_idx.tolist()}")

Choosing a CV Strategy

Tips

• 5-fold or 10-fold are common choices
• Use stratified for classification
• Always shuffle=True except for time series
• Use a pipeline to prevent data leakage
• Watch the std — the smaller it is, the more stable the model

Next chapter: Hyperparameter Tuning, how to find the best parameters for your model.

Next: Hyperparameter Tuning →