In information geometry, a divergence is a kind of statistical distance: a binary function which establishes the separation from one probability distribution to another on a statistical manifold.

The simplest divergence is squared Euclidean distance (SED), and divergences can be viewed as generalizations of SED. The other most important divergence is relative entropy (also called Kullback–Leibler divergence), which is central to information theory. There are numerous other specific divergences and classes of divergences, notably f-divergences and Bregman divergences (see § Examples).

Given a differentiable manifold ${\displaystyle M}$ of dimension ${\displaystyle n}$, a divergence on ${\displaystyle M}$ is a ${\displaystyle C^{2}}$-function ${\displaystyle D:M\times M\to [0,\infty )}$ satisfying:

In applications to statistics, the manifold ${\displaystyle M}$ is typically the space of parameters of a parametric family of probability distributions.

Condition 3 means that ${\displaystyle D}$ defines an inner product on the tangent space ${\displaystyle T_{p}M}$ for every ${\displaystyle p\in M}$. Since ${\displaystyle D}$ is ${\displaystyle C^{2}}$ on ${\displaystyle M}$, this defines a Riemannian metric ${\displaystyle g}$ on ${\displaystyle M}$.

Locally at ${\displaystyle p\in M}$, we may construct a local coordinate chart with coordinates ${\displaystyle x}$, then the divergence is $${\displaystyle D(x(p),x(p)+dx)=\textstyle {\frac {1}{2}}dx^{T}g_{p}(x)dx+O(|dx|^{3})}$$where ${\displaystyle g_{p}(x)}$ is a matrix of size ${\displaystyle n\times n}$. It is the Riemannian metric at point ${\displaystyle p}$ expressed in coordinates ${\displaystyle x}$.

Dimensional analysis of condition 3 shows that divergence has the dimension of squared distance.

The dual divergence ${\displaystyle D^{*}}$ is defined as