In mathematical statistics, the Kullback–Leibler (KL) divergence (also called relative entropy and I-divergence), denoted ${\displaystyle D_{\text{KL}}(P\parallel Q)}$, is a type of statistical distance: a measure of how much an approximating probability distribution Q is different from a true probability distribution P. Mathematically, it is defined as

$${\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in {\mathcal {X}}}P(x)\,\log {\frac {P(x)}{Q(x)}}{\text{.}}}$$

A simple interpretation of the KL divergence of P from Q is the expected excess surprisal from using the approximation Q instead of P when the actual is P. While it is a measure of how different two distributions are and is thus a distance in some sense, it is not actually a metric, which is the most familiar and formal type of distance. In particular, it is not symmetric in the two distributions (in contrast to variation of information), and does not satisfy the triangle inequality. Instead, in terms of information geometry, it is a type of divergence, a generalization of squared distance, and for certain classes of distributions (notably an exponential family), it satisfies a generalized Pythagorean theorem (which applies to squared distances).

Relative entropy is always a non-negative real number, with value 0 if and only if the two distributions in question are identical. It has diverse applications, both theoretical, such as characterizing the relative (Shannon) entropy in information systems, randomness in continuous time-series, and information gain when comparing statistical models of inference; and practical, such as applied statistics, fluid mechanics, neuroscience, bioinformatics, and machine learning.

Consider two probability distributions, a true P and an approximating Q. Often, P represents the data, the observations, or a measured probability distribution and distribution Q represents instead a theory, a model, a description, or another approximation of P. However, sometimes the true distribution P represents a model and the approximating distribution Q represents (simulated) data that are intended to match the true distribution. The Kullback–Leibler divergence ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is then interpreted as the average difference of the number of bits required for encoding samples of P using a code optimized for Q rather than one optimized for P.

Note that the roles of P and Q can be reversed in some situations where that is easier to compute and the goal is to minimize ${\displaystyle D_{\text{KL}}(P\parallel Q)}$, such as with the expectation–maximization algorithm (EM) and evidence lower bound (ELBO) computations. This role-reversal approach exploits that ${\displaystyle D_{\text{KL}}(P\parallel Q)=0}$ if and only if ${\displaystyle D_{\text{KL}}(Q\parallel P)=0}$ and that, in many cases, reducing one has the effect of reducing the other.

The relative entropy was introduced by Solomon Kullback and Richard Leibler in Kullback & Leibler (1951) as "the mean information for discrimination between ${\displaystyle H_{1}}$ and ${\displaystyle H_{2}}$ per observation from ${\displaystyle \mu _{1}}$", where one is comparing two probability measures ${\displaystyle \mu _{1},\mu _{2}}$, and ${\displaystyle H_{1},H_{2}}$ are the hypotheses that one is selecting from measure ${\displaystyle \mu _{1},\mu _{2}}$ (respectively). They denoted this by ${\displaystyle I(1:2)}$, and defined the "'divergence' between ${\displaystyle \mu _{1}}$ and ${\displaystyle \mu _{2}}$" as the symmetrized quantity ${\displaystyle J(1,2)=I(1:2)+I(2:1)}$, which had already been defined and used by Harold Jeffreys in 1948. In Kullback (1959), the symmetrized form is again referred to as the "divergence", and the relative entropies in each direction are referred to as a "directed divergences" between two distributions; Kullback preferred the term discrimination information. The term "divergence" is in contrast to a distance (metric), since the symmetrized divergence does not satisfy the triangle inequality. Numerous references to earlier uses of the symmetrized divergence and to other statistical distances are given in Kullback (1959, pp. 6–7, §1.3 Divergence). The asymmetric "directed divergence" has come to be known as the Kullback–Leibler divergence, while the symmetrized "divergence" is now referred to as the Jeffreys divergence.

For discrete probability distributions P and Q defined on the same sample space, ${\displaystyle {\mathcal {X}}}$, the relative entropy from Q to P is defined to be