Harmonic mean estimators
In connection with my recent talk at MaxEnt 2009 and older talks and posts, I have just written with Darren Wraith a very short note on some computational methods used for approximating Bayes factors. This note has now been arXived and it is strongly based on the slides for this talk. Nonetheless, I think there is an interesting point made in the section on harmonic mean estimators, namely that (a) using an MCMC output, it is easy to determine highest posterior density regions by looking directly at the numerical value of the posterior (up to a constant) and (b) those HPD regions can be used to construct pseudo-proposals whose support is one of those regions. This cancels the infinite variance difficulty rightly stressed recently by Radford Neal. In short, the fact that
![\mathbb{E}\left[ \dfrac{\varphi(\theta)}{\pi(\theta) L(\theta|x)} \mid x \right] = \dfrac{1}{m(x)}](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5Cleft%5B+%5Cdfrac%7B%5Cvarphi%28%5Ctheta%29%7D%7B%5Cpi%28%5Ctheta%29+L%28%5Ctheta%7Cx%29%7D+%5Cmid+++x+%5Cright%5D+%3D+%5Cdfrac%7B1%7D%7Bm%28x%29%7D&bg=000000&fg=B0B0B0&s=0&c=20201002)
holds for any density φ is of interest for approximating m(x) and therefore Bayes factors only when φ is such that the corresponding importance sampling estimator has a finite variance. The infamous but common choice φ=π is rarely appropriate in this regard. However, if one uses the output of an MCMC sampler to determine an empirical highest posterior density region, restricting φ to this region or to an approximation of it—like an ellipsoid—escapes the infinite variance difficulties.
November 28, 2011 at 12:12 am
[…] the exponential and average the sample to the fourth power… In this specific situation, the (dreaded) harmonic mean estimator has a finite variance! To […]
November 7, 2011 at 12:13 am
[…] like the posterior predictive p-value, residual values, cross-validation (with, once again!, the dreaded harmonic mean estimator! and, once again, Geisser and Eddy’s conditional predictive ordinates), keeping the […]
October 31, 2011 at 12:15 am
[…] estimate to approximate these cross-validation predictive densities, which indeed amount to using the dreadful harmonic mean estimate!!! They apparently missed Radford’s blog on the issue. (Chapter 4 is actually the central […]
January 8, 2011 at 12:13 am
[…] preliminary simulation from the posterior distribution. The nonparametric estimate is entered in an harmonic mean representation we previously exploited in our HPD proposal for evidence […]
December 8, 2010 at 7:37 am
[…] the resulting Bayesian procedures, including evidence, the Savage-Dickey paradox, nested sampling, harmonic mean estimators, and […]
June 27, 2010 at 12:10 am
[…] There is therefore a connection with our proposal (made with Jean-Michel Marin) of considering an HPD region for excluding the tails of the likelihood, even though the set of integration is defined as […]
January 21, 2010 at 12:04 am
[…] too long). It is based upon the harmonic mean estimator of the marginal (already discussed in this post), namely […]
November 13, 2009 at 12:13 am
[…] papers written by astronomers have been recently posted on arXiv about (new) ways to approximate evidence. Since they both perceive those approximations as some advanced form of quadrature, they are close […]
October 14, 2009 at 6:00 am
[…] as already shown in the entry on Savage-Dickey posted a few days ago. In this same example, the harmonic mean estimator of the marginal is doing surprisingly well, again because of the fit of the […]
September 30, 2009 at 12:07 am
[…] with Jean-Michel Marin, comparing the Dickey-Savage-like approximation to the Bayes factor with an harmonic mean version and the unbeatable Chib’s […]