Today, provided the Air France strike let me fly to Birmingham airport!, I am back at Gregynog Hall, Wales, for the weekend conference organised there every year by some Welsh and English statistics departments, including Warwick. Looking forward to the relaxed gathering in the glorious Welsh countryside (and hoping that my knee will have sufficiently recovered for some trail running around Gregynog Hall…!) Here are the slides of the talk I will present tomorrow:
Archive for Wales
the invasion of the stochastic gradients
Posted in Statistics with tags approximate inference, arXiv, Euler discretisation, population Monte Carlo, RKHS, scalable MCMC, stochastic gradient, stochastic gradient descent, variational Bayes methods, Wales on May 10, 2017 by xi'anWithin the same day, I spotted three submissions to arXiv involving stochastic gradient descent, that I briefly browsed on my trip back from Wales:
- Stochastic Gradient Descent as Approximate Bayesian inference, by Mandt, Hoffman, and Blei, where this technique is used as a type of variational Bayes method, where the minimum Kullback-Leibler distance to the true posterior can be achieved. Rephrasing the [scalable] MCMC algorithm of Welling and Teh (2011) as such an approximation.
- Further and stronger analogy between sampling and optimization: Langevin Monte Carlo and gradient descent, by Arnak Dalalyan, which establishes a convergence of the uncorrected Langevin algorithm to the right target distribution in the sense of the Wasserstein distance. (Uncorrected in the sense that there is no Metropolis step, meaning this is a Euler approximation.) With an extension to the noisy version, when the gradient is approximated eg by subsampling. The connection with stochastic gradient descent is thus tenuous, but Arnak explains the somewhat disappointing rate of convergence as being in agreement with optimisation rates.
- Stein variational adaptive importance sampling, by Jun Han and Qiang Liu, which relates to our population Monte Carlo algorithm, but as a non-parametric version, using RKHS to represent the transforms of the particles at each iteration. The sampling method follows two threads of particles, one that is used to estimate the transform by a stochastic gradient update, and another one that is used for estimation purposes as in a regular population Monte Carlo approach. Deconstructing into those threads allows for conditional independence that makes convergence easier to establish. (A problem we also hit when working on the AMIS algorithm.)
Why should I be Bayesian when my model is wrong?
Posted in Books, pictures, Running, Statistics, Travel, University life with tags all models are wrong, Bayesian foundations, cross validated, Cymru, Gregynog, misspecified model, Powys, Spring, statistical inference, Wales on May 9, 2017 by xi'an
Guillaume Dehaene posted the above question on X validated last Friday. Here is an except from it:
However, as everybody knows, assuming that my model is correct is fairly arrogant: why should Nature fall neatly inside the box of the models which I have considered? It is much more realistic to assume that the real model of the data p(x) differs from p(x|θ) for all values of θ. This is usually called a “misspecified” model.
My problem is that, in this more realistic misspecified case, I don’t have any good arguments for being Bayesian (i.e: computing the posterior distribution) versus simply computing the Maximum Likelihood Estimator.
Indeed, according to Kleijn, v.d Vaart (2012), in the misspecified case, the posterior distribution converges as n→∞ to a Dirac distribution centred at the MLE but does not have the correct variance (unless two values just happen to be same) in order to ensure that credible intervals of the posterior match confidence intervals for θ.
Which is a very interesting question…that may not have an answer (but that does not make it less interesting!)
A few thoughts about that meme that all models are wrong: (resonating from last week discussion):
- While the hypothetical model is indeed almost invariably and irremediably wrong, it still makes sense to act in an efficient or coherent manner with respect to this model if this is the best one can do. The resulting inference produces an evaluation of the formal model that is the “closest” to the actual data generating model (if any);
- There exist Bayesian approaches that can do without the model, a most recent example being the papers by Bissiri et al. (with my comments) and by Watson and Holmes (which I discussed with Judith Rousseau);
- In a connected way, there exists a whole branch of Bayesian statistics dealing with M-open inference;
- And yet another direction I like a lot is the SafeBayes approach of Peter Grünwald, who takes into account model misspecification to replace the likelihood with a down-graded version expressed as a power of the original likelihood.
- The very recent Read Paper by Gelman and Hennig addresses this issue, albeit in a circumvoluted manner (and I added some comments on my blog).
- In a sense, Bayesians should be the least concerned among statisticians and modellers about this aspect since the sampling model is to be taken as one of several prior assumptions and the outcome is conditional or relative to all those prior assumptions.
Gregynog #2 [jatp]
Posted in Kids, pictures, Running, Statistics, Travel, University life with tags Aberystwyth, Britain, Charles Hanbury-Tracy, concrete, Dennis Lindley, Gregynog, Gregynog Statistical Conference, jatp, Powys, seminar, Tregynon, University of Wales, University of Warwick, Wales on April 26, 2017 by xi'an
Gregynog [jatp]
Posted in pictures, Running, Statistics, Travel, University life with tags Britain, Cymru, Great-Britain, Gregynog, jatp, Montgomeryshire, Powys, University of Wales, Wales on April 24, 2017 by xi'an
