K-Nearest Neighbors

K-Nearest Neighbors (KNN) is a simple supervised learning algorithm that classifies new data based on its similarity to the nearest data points. There is no training process — KNN stores the entire dataset (lazy learning).

How It Works

1. Choose the value of k (the number of nearest neighbors)
2. Compute the distance from the new data point to all points in the training set
3. Sort them by distance
4. Take the k nearest neighbors
5. Majority vote for classification (or the average for regression)

Distance Metrics

Euclidean Distance (most common)

d = √(Σ(xi - yi)²)

Manhattan Distance

d = Σ|xi - yi|

Minkowski Distance

A generalization of Euclidean and Manhattan.

from scipy.spatial.distance import euclidean, manhattan
import numpy as np

point_a = np.array([1, 2, 3])
point_b = np.array([4, 5, 6])

print(f"Euclidean: {euclidean(point_a, point_b):.4f}")
print(f"Manhattan: {manhattan(point_a, point_b):.4f}")

Choosing the Value of K

from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

k_values = [1, 3, 5, 7, 9, 11]
for k in k_values:
    model = KNeighborsClassifier(n_neighbors=k)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    acc = accuracy_score(y_test, y_pred)
    print(f"K={k:2d} → Accuracy: {acc:.4f}")

Feature Scaling: Very Important!

KNN is distance-based, so features with a large scale will dominate the calculation. Scaling must be applied.

from sklearn.preprocessing import StandardScaler, MinMaxScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Without scaling
model_no_scale = KNeighborsClassifier(n_neighbors=5)
model_no_scale.fit(X_train, y_train)
print(f"Without scaling: {model_no_scale.score(X_test, y_test):.4f}")

# With scaling
model_scaled = KNeighborsClassifier(n_neighbors=5)
model_scaled.fit(X_train_scaled, y_train)
print(f"With scaling: {model_scaled.score(X_test_scaled, y_test):.4f}")

Example: Wine Dataset Classification

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline

wine = load_wine()
X_train, X_test, y_train, y_test = train_test_split(
    wine.data, wine.target, test_size=0.2, random_state=42
)

pipeline = make_pipeline(
    StandardScaler(),
    KNeighborsClassifier(n_neighbors=5)
)

pipeline.fit(X_train, y_train)
print(f"Accuracy: {pipeline.score(X_test, y_test):.4f}")

Pros and Cons

Weighted KNN

Giving more weight to closer neighbors:

model_weighted = KNeighborsClassifier(
    n_neighbors=5,
    weights='distance'
)
model_weighted.fit(X_train_scaled, y_train)
print(f"Weighted KNN Accuracy: {model_weighted.score(X_test_scaled, y_test):.4f}")

The next chapter covers Support Vector Machine, a powerful algorithm for classification based on the concepts of hyperplanes and margins.

Next: Support Vector Machine →