Evaluation Metrics

Evaluation metrics are how we measure how well a machine learning model performs. Choosing the right metric depends heavily on the type of problem (regression or classification).

Regression Metrics

MSE (Mean Squared Error)

from sklearn.metrics import mean_squared_error
import numpy as np

y_true = np.array([3, -0.5, 2, 7])
y_pred = np.array([2.5, 0.0, 2, 8])

mse = mean_squared_error(y_true, y_pred)
print(f"MSE: {mse:.4f}")

RMSE (Root Mean Squared Error)

rmse = np.sqrt(mean_squared_error(y_true, y_pred))
print(f"RMSE: {rmse:.4f}")

MAE (Mean Absolute Error)

from sklearn.metrics import mean_absolute_error

mae = mean_absolute_error(y_true, y_pred)
print(f"MAE: {mae:.4f}")

R² Score

from sklearn.metrics import r2_score

r2 = r2_score(y_true, y_pred)
print(f"R²: {r2:.4f}")

Comparing Regression Metrics

Metric	Formula	Interpretation
MSE	Σ(y-ŷ)²/n	Penalizes large errors
RMSE	√MSE	Same unit as the target
MAE	Σ\|y-ŷ\|/n	Robust to outliers
R²	1 - SS_res/SS_tot	1=perfect, 0=average

Classification Metrics

Accuracy

from sklearn.metrics import accuracy_score

y_true = [0, 1, 1, 0, 1, 1]
y_pred = [0, 1, 0, 0, 1, 1]

acc = accuracy_score(y_true, y_pred)
print(f"Accuracy: {acc:.4f}")

Precision, Recall, F1-Score

from sklearn.metrics import precision_score, recall_score, f1_score

precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)

print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1-Score: {f1:.4f}")

Classification Report

from sklearn.metrics import classification_report
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print(classification_report(y_test, y_pred, target_names=iris.target_names))

When to Use Which Metric?

Regression

Metric	When to Use
MSE/RMSE	Large errors are very undesirable
MAE	Many outliers, errors must be robust
R²	You want to know the proportion of variance explained

Classification

Metric	When to Use
Accuracy	Balanced dataset
Precision	Minimizing false positives (spam filter)
Recall	Minimizing false negatives (cancer detection)
F1	Imbalanced dataset, precision-recall trade-off
AUC-ROC	Threshold-independent evaluation

from sklearn.metrics import roc_auc_score, roc_curve
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)
y_prob = model.predict_proba(X_test)[:, 1]

auc = roc_auc_score(y_test, y_prob)
print(f"AUC-ROC: {auc:.4f}")

Multi-Class Evaluation

from sklearn.metrics import confusion_matrix, classification_report
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(
    digits.data, digits.target, test_size=0.2, random_state=42
)

model = SVC()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print(f"Accuracy: {model.score(X_test, y_test):.4f}")
print(f"\nConfusion Matrix:\n{confusion_matrix(y_test, y_pred)}")

Next chapter: Confusion Matrix, a detailed visualization of classification performance.

Next: Confusion Matrix →