small-noise analysis and symmetrization of implicit Monte Carlo samplers
I read this arXived paper of Goodman, Lin and Morzfeld on a (long) train trip to the North of Paris, paper that deals with asymptotics of importance sampling, but I still have trouble about its impact on statistical (Bayesian) problems. The paper is indeed mostly about asymptotics of symmetrized versions of importance sampling, but there is a noise parameter ε that does not make sense [to me] when I try to apply it to statistical problems…
The first symmetrization sounds like a special case of the generic group operator of Kong et al., 2003, in their Read Paper. Namely that if one takes advantage in potential symmetries in the importance distribution to multiply the number of proposed values and integrates this choice into a symmetrized importance distribution, hence using a sort of Rao-Blackwellisation, the effective sample size gets better… Beside symmetry, another transform of importance sampling proposals is rescaling so that the importance density at the proposal equates the target density at the rescaled version. (Those notions of symmetry and importance proposal require some specific knowledge about the mode and Hessian of the target.) This version is called a random map, even though I have trouble spotting the randomness in the transform. The third notion in this paper is the small noise version when a small ε is introduced along with the rescaled target
which is used to evaluate the efficiency of those different transforms as ε goes to zero. Symmetrized versions improve the effective sample size by a power of ε. Namely from ε to ε² . But I still fail to see why this matters in the original problem when ε=1.
January 2, 2015 at 3:10 am
You ask a few questions that the author can answer, so here goes (all from the paper, but clearly not very clearly):
The small noise expansion is for situations where the distribution is approximately normal. The “small noise” parameter measures the degree of non-gaussianity. It’s particularly useful for SDE applications.
The theory says symmetrization helps as epsilon -> 0. Numerical experiments show that it can help on problems with epsilon = 1.
We thought about using other symmetries besides the +- symmetry of Gaussians. But all the other symmetries we tried need exponentially many evaluations (as a function of the dimension) to improve the order of approximation. The +- symmetry uses two evaluations to improve by one order.