In statistical classification, the Bayes classifier is the classifier having the smallest probability of misclassification of all classes using the same set of features.

Suppose a pair ${\displaystyle (X,Y)}$ takes values in ${\displaystyle \mathbb {R} ^{d}\times \{1,2,\dots ,K\}}$, where ${\displaystyle Y}$ is the class label of an element whose features are given by ${\displaystyle X}$. Assume that the conditional distribution of X, given that the label Y takes the value r is given by $${\displaystyle (X\mid Y=r)\sim P_{r}\quad {\text{for}}\quad r=1,2,\dots ,K}$$ where "${\displaystyle \sim }$" means "is distributed as", and where ${\displaystyle P_{r}}$ denotes a probability distribution.

A classifier is a rule that assigns to an observation X=x a guess or estimate of what the unobserved label Y=r actually was. In theoretical terms, a classifier is a measurable function ${\displaystyle C:\mathbb {R} ^{d}\to \{1,2,\dots ,K\}}$, with the interpretation that C classifies the point x to the class C(x). The probability of misclassification, or risk, of a classifier C is defined as $${\displaystyle {\mathcal {R}}(C)=\operatorname {P} \{C(X)\neq Y\}.}$$

The Bayes classifier is $${\displaystyle C^{\text{Bayes}}(x)={\underset {r\in \{1,2,\dots ,K\}}{\operatorname {argmax} }}\operatorname {P} (Y=r\mid X=x).}$$

In practice, as in most of statistics, the difficulties and subtleties are associated with modeling the probability distributions effectively—in this case, ${\displaystyle \operatorname {P} (Y=r\mid X=x)}$. The Bayes classifier is a useful benchmark in statistical classification.

The excess risk of a general classifier ${\displaystyle C}$ (possibly depending on some training data) is defined as ${\displaystyle {\mathcal {R}}(C)-{\mathcal {R}}(C^{\text{Bayes}}).}$ Thus this non-negative quantity is important for assessing the performance of different classification techniques. A classifier is said to be consistent if the excess risk converges to zero as the size of the training data set tends to infinity.

Considering the components ${\displaystyle x_{i}}$ of ${\displaystyle x}$ to be mutually independent, we get the naive Bayes classifier, where $${\displaystyle C^{\text{Bayes}}(x)={\underset {r\in \{1,2,\dots ,K\}}{\operatorname {argmax} }}\operatorname {P} (Y=r)\prod _{i=1}^{d}P_{r}(x_{i}).}$$

Proof that the Bayes classifier is optimal and Bayes error rate is minimal proceeds as follows.