Bhattacharyya distance

In statistics, the Bhattacharyya distance is a quantity which represents a notion of similarity between two probability distributions. It is closely related to the Bhattacharyya coefficient, which is a measure of the amount of overlap between two statistical samples or populations.

It is not a metric, despite being named a "distance", since it does not obey the triangle inequality.

Both the Bhattacharyya distance and the Bhattacharyya coefficient are named after Anil Kumar Bhattacharyya, a statistician who worked in the 1930s at the Indian Statistical Institute. He has developed this through a series of papers. He developed the method to measure the distance between two non-normal distributions and illustrated this with the classical multinomial populations, this work despite being submitted for publication in 1941, appeared almost five years later in Sankhya. Consequently, Professor Bhattacharyya started working toward developing a distance metric for probability distributions that are absolutely continuous with respect to the Lebesgue measure and published his progress in 1942, at Proceedings of the Indian Science Congress and the final work has appeared in 1943 in the Bulletin of the Calcutta Mathematical Society.

For probability distributions ${\displaystyle P}$ and ${\displaystyle Q}$ on the same discrete domain ${\displaystyle {\mathcal {X}}}$, the Bhattacharyya distance is defined as $${\displaystyle D_{B}(P,Q)=-\ln \left(BC(P,Q)\right)}$$ where $${\displaystyle BC(P,Q)=\sum _{x\in {\mathcal {X}}}{\sqrt {P(x)Q(x)}}}$$ is the Bhattacharyya coefficient for discrete probability distributions.

For continuous probability distributions, with ${\displaystyle P(dx)=p(x)dx}$ and ${\displaystyle Q(dx)=q(x)dx}$ where ${\displaystyle p(x)}$ and ${\displaystyle q(x)}$ are the probability density functions, the Bhattacharyya coefficient is defined as $${\displaystyle BC(P,Q)=\int _{\mathcal {X}}{\sqrt {p(x)q(x)}}\,dx.}$$

More generally, given two probability measures ${\displaystyle P,Q}$ on a measurable space ${\displaystyle ({\mathcal {X}},{\mathcal {B}})}$, let ${\displaystyle \lambda }$ be a (sigma finite) measure such that ${\displaystyle P}$ and ${\displaystyle Q}$ are absolutely continuous with respect to ${\displaystyle \lambda }$ i.e. such that ${\displaystyle P(dx)=p(x)\lambda (dx)}$, and ${\displaystyle Q(dx)=q(x)\lambda (dx)}$ for probability density functions ${\displaystyle p,q}$ with respect to ${\displaystyle \lambda }$ defined ${\displaystyle \lambda }$-almost everywhere. Such a measure, even such a probability measure, always exists, e.g. ${\displaystyle \lambda ={\tfrac {1}{2}}(P+Q)}$. Then define the Bhattacharyya measure on ${\displaystyle ({\mathcal {X}},{\mathcal {B}})}$ by $${\displaystyle {\begin{aligned}bc(dx|P,Q)&={\sqrt {p(x)q(x)}}\,\lambda (dx)\\[1ex]&={\sqrt {{\frac {P(dx)}{\lambda (dx)}}(x){\frac {Q(dx)}{\lambda (dx)}}(x)}}\lambda (dx).\end{aligned}}}$$ It does not depend on the measure ${\displaystyle \lambda }$, for if we choose a measure ${\displaystyle \mu }$ such that ${\displaystyle \lambda }$ and an other measure choice ${\displaystyle \lambda '}$ are absolutely continuous i.e. ${\displaystyle \lambda =l(x)\mu }$ and ${\displaystyle \lambda '=l'(x)\mu }$, then $${\displaystyle {\begin{alignedat}{3}P(dx)&=p(x)\lambda (dx)&&={}&&p'(x)\lambda '(dx)\\&=p(x)l(x)\mu (dx)&&={}&&p'(x)l'(x)\mu (dx),\end{alignedat}}}$$ and similarly for ${\displaystyle Q}$. We then have $${\displaystyle {\begin{aligned}bc(dx|P,Q)&={\sqrt {p(x)q(x)}}\,\lambda (dx)={\sqrt {p(x)q(x)}}\,l(x)\mu (x)\\&={\sqrt {p(x)l(x)q(x)\,l(x)}}\mu (dx)={\sqrt {p'(x)l'(x)q'(x)l'(x)}}\,\mu (dx)\\&={\sqrt {p'(x)q'(x)}}\,\lambda '(dx).\end{aligned}}}$$ We finally define the Bhattacharyya coefficient $${\displaystyle BC(P,Q)=\int _{\mathcal {X}}bc(dx|P,Q)=\int _{\mathcal {X}}{\sqrt {p(x)q(x)}}\,\lambda (dx).}$$ By the above, the quantity ${\displaystyle BC(P,Q)}$ does not depend on ${\displaystyle \lambda }$, and by the Cauchy inequality ${\displaystyle 0\leq BC(P,Q)\leq 1}$. Using ${\displaystyle P(dx)=p(x)\lambda (dx)}$, and ${\displaystyle Q(dx)=q(x)\lambda (dx)}$, $${\displaystyle BC(P,Q)=\int _{\mathcal {X}}{\sqrt {\frac {p(x)}{q(x)}}}Q(dx)=\int _{\mathcal {X}}{\sqrt {\frac {P(dx)}{Q(dx)}}}Q(dx)=E_{Q}\left[{\sqrt {\frac {P(dx)}{Q(dx)}}}\right]}$$

Let ${\displaystyle p\sim {\mathcal {N}}(\mu _{p},\sigma _{p}^{2})}$, ${\displaystyle q\sim {\mathcal {N}}(\mu _{q},\sigma _{q}^{2})}$, where ${\displaystyle {\mathcal {N}}(\mu ,\sigma ^{2})}$ is the normal distribution with mean ${\displaystyle \mu }$ and variance ${\displaystyle \sigma ^{2}}$; then

$${\displaystyle D_{B}(p,q)={\frac {1}{4}}{\frac {(\mu _{p}-\mu _{q})^{2}}{\sigma _{p}^{2}+\sigma _{q}^{2}}}+{\frac {1}{2}}\ln \left({\frac {\sigma _{p}^{2}+\sigma _{q}^{2}}{2\sigma _{p}\sigma _{q}}}\right).}$$