In statistical learning theory, the principle of empirical risk minimization defines a family of learning algorithms based on evaluating performance over a known and fixed dataset. The core idea is based on an application of the law of large numbers; more specifically, we cannot know exactly how well a predictive algorithm will work in practice (i.e. the "true risk") because we do not know the true distribution of the data, but we can instead estimate and optimize the performance of the algorithm on a known set of training data. The performance over the known set of training data is referred to as the "empirical risk".

The following situation is a general setting of many supervised learning problems. There are two spaces of objects ${\displaystyle X}$ and ${\displaystyle Y}$ and we would like to learn a function ${\displaystyle \ h:X\to Y}$ (often called hypothesis) which outputs an object ${\displaystyle y\in Y}$, given ${\displaystyle x\in X}$. To do so, there is a training set of ${\displaystyle n}$ examples ${\displaystyle \ (x_{1},y_{1}),\ldots ,(x_{n},y_{n})}$ where ${\displaystyle x_{i}\in X}$ is an input and ${\displaystyle y_{i}\in Y}$ is the corresponding response that is desired from ${\displaystyle h(x_{i})}$.

To put it more formally, assuming that there is a joint probability distribution ${\displaystyle P(x,y)}$ over ${\displaystyle X}$ and ${\displaystyle Y}$, and that the training set consists of ${\displaystyle n}$ instances ${\displaystyle \ (x_{1},y_{1}),\ldots ,(x_{n},y_{n})}$ drawn i.i.d. from ${\displaystyle P(x,y)}$. The assumption of a joint probability distribution allows for the modelling of uncertainty in predictions (e.g. from noise in data) because ${\displaystyle y}$ is not a deterministic function of ${\displaystyle x}$, but rather a random variable with conditional distribution ${\displaystyle P(y|x)}$ for a fixed ${\displaystyle x}$.

It is also assumed that there is a non-negative real-valued loss function ${\displaystyle L({\hat {y}},y)}$ which measures how different the prediction ${\displaystyle {\hat {y}}}$ of a hypothesis is from the true outcome ${\displaystyle y}$. For classification tasks, these loss functions can be scoring rules. The risk associated with hypothesis ${\displaystyle h(x)}$ is then defined as the expectation of the loss function:

A loss function commonly used in theory is the 0-1 loss function: ${\displaystyle L({\hat {y}},y)={\begin{cases}1&{\mbox{ if }}\quad {\hat {y}}\neq y\\0&{\mbox{ if }}\quad {\hat {y}}=y\end{cases}}}$.

The ultimate goal of a learning algorithm is to find a hypothesis ${\displaystyle h^{*}}$ among a fixed class of functions ${\displaystyle {\mathcal {H}}}$ for which the risk ${\displaystyle R(h)}$ is minimal:

For classification problems, the Bayes classifier is defined to be the classifier minimizing the risk defined with the 0–1 loss function.

In general, the risk ${\displaystyle R(h)}$ cannot be computed because the distribution ${\displaystyle P(x,y)}$ is unknown to the learning algorithm. However, given a sample of iid training data points, we can compute an estimate, called the empirical risk, by computing the average of the loss function over the training set; more formally, computing the expectation with respect to the empirical measure: