«
»

Bhattacharyya distance versus Kullback-Leibler divergence

Another question I picked on Cross Validated during the Yule break is about the connection between the Bhattacharyya distance and the Kullback-Leibler divergence, i.e.,

d_B(p,q)=-\log\left\{\int\sqrt{p(x)q(x)}\,\text{d}x\right\}

and

d_{KL}(p\|q)=\int\log\left\{{q(x)}\big/{p(x)}\right\}\,p(x)\,\text{d}x

Although this Bhattacharyya distance sounds close to the Hellinger distance,

d_H(p,q)=\left\{1-\int\sqrt{p(x)q(x)}\,\text{d}x\right\}^{1/2}

the ordering I got by a simple Jensen inequality is

d_{KL}(p\|q)\ge2d_B(p,q)\ge2d_H(p,q)^2\,.

and I wonder how useful this ordering could be…

This entry was posted on January 10, 2015 at 12:15 am and is filed under Books, Kids, Statistics with tags , , , , . You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

One Response to “Bhattacharyya distance versus Kullback-Leibler divergence”

  1. Dan Simpson's avatar
    Dan Simpson Says:
    January 10, 2015 at 12:40 am

    They’re all very similar. The wiki page for Bhattacharyya distance gives the precise (equality) relationship between d_B and d_H (d_B is not a metric). So I guess that the Bhattacharyya distance and the Hellinger distance define equivalent topologies.

    The ordering is that KLD is (topologically) stronger than Hellinger which is stronger than total variation. And convergence in the Hellinger metric implies that L2 funcitonals converge.

    Reply

Leave a Reply

Discover more from Xi'an's Og

Subscribe now to keep reading and get access to the full archive.

Continue reading