Naive Bayes

Gaussian Naive Bayes classifier.

Implements the Gaussian Naive Bayes algorithm for classification. Assumes features follow a Gaussian (normal) distribution.

Algorithm: 1. Calculate mean and variance for each feature per class 2. For prediction, calculate likelihood using Gaussian PDF 3. Apply Bayes' theorem to get posterior probabilities 4. Predict class with highest posterior probability

Like scikit-learn, varSmoothing is a fraction of the largest per-feature variance of the whole training set; that value is added to every class variance. If every feature is constant, varSmoothing itself is added.

Time Complexity: - Training: O(n * d) where n=samples, d=features - Prediction: O(k * d) per sample where k=classes

Bernoulli Naive Bayes classifier.

Like MultinomialNB, this classifier is suitable for discrete data. The difference is that while MultinomialNB works with occurrence counts, BernoulliNB is designed for binary/boolean features. A feature counts as present when its value is strictly greater than the binarize threshold. With binarize: null the threshold is 0, so 0/1 data is used as is. Absent features contribute log(1 - p) to the score, which distinguishes this model from MultinomialNB.

Categorical Naive Bayes classifier.

Suitable for classification with discrete/categorical features. Each feature is assumed to be generated from a categorical distribution.

Unlike MultinomialNB (which models feature counts) or BernoulliNB (binary), CategoricalNB models each feature as having its own set of categories.

Algorithm: 1. For each feature and each class, compute the probability of each category using smoothed maximum likelihood estimates. 2. For prediction, compute the log posterior for each class.

Categories: a feature's categories are the distinct values seen in training. When every training value of a feature is a non-negative integer (ordinal codes, as scikit-learn requires), the categories are 0..max, so codes that never appear still count towards the smoothing denominator; minCategories raises that count further. A category not seen for a class, or not seen at all, gets probability alpha / (count_c + alpha * n_categories).

Complement Naive Bayes classifier.

An adaptation of the standard Multinomial Naive Bayes that is particularly suited for imbalanced datasets. Instead of computing the likelihood from samples of a class, it uses the complement (all samples NOT in the class).

Algorithm (as in scikit-learn): 1. For each class c, count every feature over the samples of all other classes and smooth the counts with alpha. 2. Take w_cj = -log(theta_cj), where theta_cj is the smoothed complement frequency of feature j. With norm: true the weights of each class are divided by their sum (the L1 norm, since all weights are positive). 3. Score class c as sum_j x_j * w_cj and predict the class with the highest score.

The class prior is only used when there is a single class, as in scikit-learn.

Reference: Rennie et al., "Tackling the Poor Assumptions of Naive Bayes Text Classifiers", ICML 2003.

Multinomial Naive Bayes classifier.

Suitable for classification with discrete features (e.g., word counts for text classification). The multinomial distribution normally requires integer feature counts, but fractional counts (e.g., TF-IDF) work in practice.

Algorithm: 1. For each class, compute the log probability of each feature using smoothed maximum likelihood estimates. 2. For prediction, compute the log posterior for each class and select the class with the highest posterior.