Likelihood function

A likelihood function (often simply called the likelihood) measures how well a statistical model explains observed data by calculating the probability of seeing that data under different parameter values of the model. It is constructed from the joint probability distribution of the random variable that (presumably) generated the observations. When evaluated on the actual data points, it becomes a function solely of the model parameters.

In maximum likelihood estimation, the model parameter(s) or argument that maximizes the likelihood function serves as a point estimate for the unknown parameter, while the Fisher information (often approximated by the likelihood's Hessian matrix at the maximum) gives an indication of the estimate's precision.

In contrast, in Bayesian statistics, the estimate of interest is the converse of the likelihood, the so-called posterior probability of the parameter given the observed data, which is calculated via Bayes' rule.

The likelihood function, parameterized by a (possibly multivariate) parameter ${\textstyle \theta }$, is usually defined differently for discrete and continuous probability distributions (a more general definition is discussed below). Given a probability density or mass function

$${\displaystyle x\mapsto f(x\mid \theta ),}$$

where ${\textstyle x}$ is a realization of the random variable ${\textstyle X}$, the likelihood function is $${\displaystyle \theta \mapsto f(x\mid \theta ),}$$ often written $${\displaystyle {\mathcal {L}}(\theta \mid x).}$$

In other words, when ${\textstyle f(x\mid \theta )}$ is viewed as a function of ${\textstyle x}$ with ${\textstyle \theta }$ fixed, it is a probability density function, and when viewed as a function of ${\textstyle \theta }$ with ${\textstyle x}$ fixed, it is a likelihood function. In the frequentist paradigm, the notation ${\textstyle f(x\mid \theta )}$ is often avoided and instead ${\textstyle f(x;\theta )}$ or ${\textstyle f(x,\theta )}$ are used to indicate that ${\textstyle \theta }$ is regarded as a fixed unknown quantity rather than as a random variable being conditioned on.

The likelihood function does not specify the probability that ${\textstyle \theta }$ is the truth, given the observed sample ${\textstyle X=x}$. Such an interpretation is a common error, with potentially disastrous consequences (see prosecutor's fallacy).