Naive Bayes

Naive Bayes is a probabilistic classifier built on Bayes' theorem: compute the probability of each class given the features, and pick the largest. It is "naive" because it assumes the features are conditionally independent given the class - an assumption that is almost never literally true - and yet it works well, trains in a single fast pass, and is the standard baseline for text classification. Its strengths are speed, small-data behavior, and a natural probability output; its weakness is that the independence assumption can badly miscalibrate when features are correlated.

The Math in One Paragraph

Bayes' theorem flips a probability: P(class | features) is proportional to P(features | class) times P(class). The "naive" step factors the feature likelihood into a product of per-feature terms - P(f1, f2 | class) = P(f1 | class) * P(f2 | class) - which makes the computation linear in the number of features instead of exponential. Each variant of Naive Bayes is a choice of distribution for those per-feature terms: Gaussian for continuous features, multinomial for word counts, Bernoulli for binary presence.

GaussianNB on Continuous Data

from sklearn.datasets import make_classification
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=1000, n_features=10, n_informative=5,
                           random_state=7)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

model = GaussianNB().fit(Xtr, ytr)
print("test acc:", round(accuracy_score(yte, model.predict(Xte)), 3))
print("class priors:", model.class_prior_.round(3))
proba = model.predict_proba(Xte)
print("probability row sums:", proba.sum(axis=1)[:3].round(6))

GaussianNB fits each feature's per-class distribution as a normal (mean and variance per class) and class prior. Fitting is closed-form - no optimization loop - so it trains instantly even on large data. predict_proba returns calibrated-enough class probabilities directly, which is why it is a common first model when the downstream consumer is a threshold, not a label.

MultinomialNB on Text

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

docs = ["the game was great and exciting",
        "what a terrible boring match",
        "amazing performance, loved every minute",
        "worst show of the season, dull",
        "fantastic play and wonderful atmosphere",
        "a pointless and forgettable affair"]
labels = ["pos", "neg", "pos", "neg", "pos", "neg"]

pipe = make_pipeline(CountVectorizer(), MultinomialNB())
pipe.fit(docs, labels)

test = ["a great and exciting game", "boring and dull"]
print(pipe.predict(test))          # ['pos' 'neg']
print(pipe.predict_proba(test).round(2))

The text pipeline - CountVectorizer (words to counts) plus MultinomialNB - is the classic spam-filter architecture, and it is still shockingly competitive on short-text classification with modest training data. The multinomial model treats word counts as draws from a per-class vocabulary distribution, which is a good fit for bag-of-words features even though word independence is plainly false.

The Smoothing Knob

# alpha = Laplace smoothing: how strongly to assume unseen words/feature values
# alpha=1 (default) adds a pseudocount to every term; smaller alpha trusts the data more
model = MultinomialNB(alpha=0.1).fit(Xtr_counts, ytr)
# for GaussianNB, var_smoothing adds a tiny fraction of the max variance to each
# feature variance, preventing zero-variance collapse on small data
model_g = GaussianNB(var_smoothing=1e-9).fit(Xtr, ytr)

Smoothing keeps the model defined when a feature value (or word) never appeared in a class during training - without it, a single unseen token would give a zero probability and annihilate the whole product. The alpha value is a mild regularizer: tune it lightly on a validation set, but its range of reasonable values is narrow.

When Naive Bayes Is the Right Tool

Use it as the fast baseline for text and high-dimensional sparse data, when training must be instant, when data is small, and when you want probabilities with a single pass.
Expect it to lose to logistic regression and trees when features are strongly correlated and the dataset is large enough for the stronger model to use that structure.
Do not use it when the feature correlations are the signal (the assumption works against you) or when calibration precision matters more than speed.

Summary

Fit GaussianNB for continuous features and the CountVectorizer-plus-MultinomialNB pipeline for text: both train in closed form, output probabilities directly, and are the right first model on small or high-dimensional-sparse data. Tune the smoothing knob lightly, understand that the independence assumption is a trade (speed and small-data robustness for miscalibration under correlation), and let the evaluation step decide whether the naive model is good enough or whether the correlated features deserve a stronger classifier.