In variational Bayesian methods, the evidence lower bound (often abbreviated ELBO, also sometimes called the variational lower bound or negative variational free energy) is a useful lower bound on the log-likelihood of some observed data.

The ELBO is useful because it provides a guarantee on the worst-case for the log-likelihood of some distribution (e.g. ${\displaystyle p(X)}$) which models a set of data. The actual log-likelihood may be higher (indicating an even better fit to the distribution) because the ELBO includes a Kullback-Leibler divergence (KL divergence) term which decreases the ELBO due to an internal part of the model being inaccurate despite good fit of the model overall. Thus improving the ELBO score indicates either improving the likelihood of the model ${\displaystyle p(X)}$ or the fit of a component internal to the model, or both, and the ELBO score makes a good loss function, e.g., for training a deep neural network to improve both the model overall and the internal component. (The internal component is ${\displaystyle q_{\phi }(\cdot |x)}$, defined in detail later in this article.)

Let ${\displaystyle X}$ and ${\displaystyle Z}$ be random variables, jointly distributed with distribution ${\displaystyle p_{\theta }}$. For example, ${\displaystyle p_{\theta }(X)}$ is the marginal distribution of ${\displaystyle X}$, and ${\displaystyle p_{\theta }(Z\mid X)}$ is the conditional distribution of ${\displaystyle Z}$ given ${\displaystyle X}$. Then, for a sample ${\displaystyle x\sim p_{\text{data}}}$, and any distribution ${\displaystyle q_{\phi }}$, the ELBO is defined as$${\displaystyle L(\phi ,\theta ;x):=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln {\frac {p_{\theta }(x,z)}{q_{\phi }(z|x)}}\right].}$$The ELBO can equivalently be written as

$${\displaystyle {\begin{aligned}L(\phi ,\theta ;x)=&\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln {}p_{\theta }(x,z)\right]+H[q_{\phi }(z|x)]\\=&\mathbb {\ln } {}\,p_{\theta }(x)-D_{KL}(q_{\phi }(z|x)||p_{\theta }(z|x)).\\\end{aligned}}}$$

In the first line, ${\displaystyle H[q_{\phi }(z|x)]}$ is the entropy of ${\displaystyle q_{\phi }}$, which relates the ELBO to the Helmholtz free energy. In the second line, ${\displaystyle \ln p_{\theta }(x)}$ is called the evidence for ${\displaystyle x}$, and ${\displaystyle D_{KL}(q_{\phi }(z|x)||p_{\theta }(z|x))}$ is the Kullback-Leibler divergence between ${\displaystyle q_{\phi }}$ and ${\displaystyle p_{\theta }}$. Since the Kullback-Leibler divergence is non-negative, ${\displaystyle L(\phi ,\theta ;x)}$ forms a lower bound on the evidence (ELBO inequality)$${\displaystyle \ln p_{\theta }(x)\geq \mathbb {\mathbb {E} } _{z\sim q_{\phi }(\cdot |x)}\left[\ln {\frac {p_{\theta }(x,z)}{q_{\phi }(z\vert x)}}\right].}$$

Suppose we have an observable random variable ${\displaystyle X}$, and we want to find its true distribution ${\displaystyle p^{*}}$. This would allow us to generate data by sampling, and estimate probabilities of future events. In general, it is impossible to find ${\displaystyle p^{*}}$ exactly, forcing us to search for a good approximation.

That is, we define a sufficiently large parametric family ${\displaystyle \{p_{\theta }\}_{\theta \in \Theta }}$ of distributions, then solve for ${\displaystyle \min _{\theta }L(p_{\theta },p^{*})}$ for some loss function ${\displaystyle L}$. One possible way to solve this is by considering small variation from ${\displaystyle p_{\theta }}$ to ${\displaystyle p_{\theta +\delta \theta }}$, and solve for ${\displaystyle L(p_{\theta },p^{*})-L(p_{\theta +\delta \theta },p^{*})=0}$. This is a problem in the calculus of variations, thus it is called the variational method.

Since there are not many explicitly parametrized distribution families (all the classical distribution families, such as the normal distribution, the Gumbel distribution, etc, are far too simplistic to model the true distribution), we consider implicitly parametrized probability distributions: