Naive Bayes

Naive Bayes is a family of classification algorithms based on Bayes' Theorem, with the "naive" assumption that each feature is independent of the others. Although this assumption is not always true, the algorithm often works very well in practice.

Bayes' Theorem

P(A|B) = P(B|A) × P(A) / P(B)

In the context of classification:

P(class|features) = P(features|class) × P(class) / P(features)

The Naive Assumption

Assumption: all features are independent of each other:

P(x1,x2,...,xn|class) = P(x1|class) × P(x2|class) × ... × P(xn|class)

Although this assumption is rarely true in reality, Naive Bayes remains effective in many cases.

Types of Naive Bayes

from sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

models = {
    'GaussianNB': GaussianNB(),
    'MultinomialNB': MultinomialNB(),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    acc = model.score(X_test, y_test)
    print(f"{name}: {acc:.4f}")

Example: Spam Detection

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

emails = [
    "Congratulations, you have won a prize drawing",
    "Meeting tomorrow at 9 AM",
    "Click this link to claim your bonus",
    "The Q3 financial report is done",
    "Big discount promo today",
    "Reminder: project presentation tomorrow",
    "You have been selected to win a car",
    "Please review the document that has been sent"
]
labels = [1, 0, 1, 0, 1, 0, 1, 0]  # 1=spam, 0=ham

pipeline = make_pipeline(CountVectorizer(), MultinomialNB())
pipeline.fit(emails, labels)

test_emails = [
    "You won the raffle! Claim now",
    "Don't forget the meeting at 3 PM"
]
prediction = pipeline.predict(test_emails)
for email, pred in zip(test_emails, prediction):
    print(f"'{email[:40]}...' → {'SPAM' if pred == 1 else 'HAM'}")

Comparing the Three Types

from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB
from sklearn.preprocessing import MinMaxScaler

wine = load_wine()
X, y = wine.data, wine.target

models = {
    'GaussianNB': GaussianNB(),
    'MultinomialNB': MultinomialNB(),
    'BernoulliNB': BernoulliNB(),
}

for name, model in models.items():
    if name == 'MultinomialNB':
        scaler = MinMaxScaler()
        X_scaled = scaler.fit_transform(X)
        scores = cross_val_score(model, X_scaled, y, cv=5)
    elif name == 'BernoulliNB':
        scaler = MinMaxScaler()
        X_scaled = scaler.fit_transform(X)
        scores = cross_val_score(model, X_scaled > 0.5, y, cv=5)
    else:
        scores = cross_val_score(model, X, y, cv=5)
    print(f"{name:15s}: {scores.mean():.4f} (+/- {scores.std():.4f})")

Pros and Cons

Practical Tips

• Use GaussianNB for continuous features
• Use MultinomialNB for text classification (TF-IDF, counts)
• Use BernoulliNB for binary/boolean features
• Handle the zero frequency problem with Laplace smoothing (alpha parameter)

In the next chapter we enter the world of unsupervised learning: K-Means Clustering.

Next: K-Means Clustering →