Archive for model choice

Bayesian model selection

Posted in Books, R, Statistics with tags , , , , , , , , , on December 8, 2010 by xi'an

Last week, I received a box of books from the International Statistical Review, for reviewing them. I thus grabbed the one whose title was most appealing to me, namely Bayesian Model Selection and Statistical Modeling by Tomohiro Ando. I am indeed interested in both the nature of testing hypotheses or more accurately of assessing models, as discussed in both my talk at the Seminar of philosophy of mathematics at Université Paris Diderot a few days ago and the post on Murray Aitkin’s alternative, and the computational aspects of the resulting Bayesian procedures, including evidence, the Savage-Dickey paradox, nested sampling, harmonic mean estimators, and more…

After reading through the book, I am alas rather disappointed. What I consider to be innovative or at least “novel” parts with comparison with existing books (like Chen, Shao and Ibrahim, 2000, which remains a reference on this topic) is based on papers written by the author over the past five years and it is mostly a sort of asymptotic Bayes analysis that I do not see as particularly Bayesian, because involving the “true” distribution of the data. The coverage of the existing literature on Bayesian model choice is often incomplete and sometimes misses the point, as discussed below. This is especially true for the computational aspects that are generally mistreated or at least not treated in a way from which a newcomer to the field would benefit. The author often takes complex econometric examples for illustration, which is nice; however, he does not pursue the details far enough for the reader to be able to replicate the study without further reading. (An example is given by the coverage of stochastic volatility in Section 4.5.1, pages 83-84.) The few exercises at the end of each chapter are rather unhelpful, often sounding rather like notes than true problems (an extreme case is Exercise 6 pages 196-197 which introduces the Metropolis-Hastings algorithm within the exercise (although it has already been defined on pages 66-67) and then asks to derive the marginal likelihood estimator. Another such exercise on page 164-165 introduces the theory of DNA microarrays and gene expression in ten lines (which are later repeated verbatim on page 227), then asks to identify marker genes responsible for a certain trait.) The overall feeling after reading this book is thus that the contribution to the field of Bayesian Model Selection and Statistical Modeling is too limited and disorganised for the book to be recommended as “helping you choose the right Bayesian model” (backcover).

Continue reading →

“Bayesian model comparison in cosmology” on-line

Posted in Statistics, University life with tags , , , , , on June 27, 2010 by xi'an

I actually missed the piece of information that our our paper “Bayesian model comparison in cosmology with Population Monte Carlo” has been accepted by Monthly Notices of the Royal Astronomical Society on March 1! The abstract if not the whole paper is available on-line as early-view since mid-April… This is my last paper published in collaboration with the cosmologists of the Ecosstat 2005-2009 ANR program. Hopefully not the end of our collaboration as this was a very fruitful experience from my viewpoint, which happened to coincide with the golden years of population Monte Carlo, just as the Misgepop ANR program launched our foray into ABC methods. (In case you are unaware of the link, Scott Sisson has a twitter page posting news on ABC methods.)

Bayes vs. SAS

Posted in Books, R, Statistics with tags , , , , , , , , , , , , , , , , , , on May 7, 2010 by xi'an

Glancing perchance at the back of my Amstat News, I was intrigued by the SAS advertisement

Bayesian Methods

  • Specify Bayesian analysis for ANOVA, logistic regression, Poisson regression, accelerated failure time models and Cox regression through the GENMOD, LIFEREG and PHREG procedures.
  • Analyze a wider variety of models with the MCMC procedure, a general purpose Bayesian analysis procedure.

and so decided to take a look at those items on the SAS website. (Some entries date back to 2006 so I am not claiming novelty in this post, just my reading through the manual!)

Even though I have not looked at a SAS program since the time in 1984 I was learning principal component and discriminant analysis by programming SAS procedures on punched cards, it seems the MCMC part is rather manageable (if you can manage SAS at all!), looking very much like a second BUGS to my bystander eyes, even to the point of including ARS algorithms! The models are defined in a BUGS manner, with priors on the side (and this includes improper priors, despite a confusing first example that mixes very large variances with vague priors for the linear model!). The basic scheme is a random walk proposal with adaptive scale or covariance matrix. (The adaptivity on the covariance matrix is slightly confusing in that the way it is described it does not seem to implement the requirements of Roberts and Rosenthal for sure convergence.) Gibbs sampling is not directly covered, although some examples are in essence using Gibbs samplers. Convergence is assessed via ca. 1995 methods à la Cowles and Carlin, including the rather unreliable Raftery and Lewis indicator, but so does Introducing Monte Carlo Methods with R, which takes advantage of the R coda package. I have not tested (!) any of the features in the MCMC procedure but judging from a quick skim through the 283 page manual everything looks reasonable enough. I wonder if anyone has ever tested a SAS program against its BUGS counterpart for efficiency comparison.

The Bayesian aspects are rather traditional as well, except for the testing issue. Indeed, from what I have read, SAS does not engage into testing and remains within estimation bounds, offering only HPD regions for variable selection without producing a genuine Bayesian model choice tool. I understand the issues with handling improper priors versus computing Bayes factors, as well as some delicate computational requirements, but this is a truly important chunk missing from the package. (Of course, the package contains a DIC (Deviance information criterion) capability, which may be seen as a substitute, but I have reservations about the relevance of DIC outside generalised linear models. Same difficulty with the posterior predictive.) As usual with SAS, the documentation is huge (I still remember the shelves after shelves of documentation volumes in my 1984 card-punching room!) and full of options and examples. Nothing to complain about. Except maybe the list of disadvantages in using Bayesian analysis:

  • It does not tell you how to select a prior. There is no correct way to choose a prior. Bayesian inferences require skills to translate prior beliefs into a mathematically formulated prior. If you do not proceed with caution, you can generate misleading results.
  • It can produce posterior distributions that are heavily influenced by the priors. From a practical point of view, it might sometimes be difficult to convince subject matter experts who do not agree with the validity of the chosen prior.
  • It often comes with a high computational cost, especially in models with a large number of parameters.

which does not say much… Since the MCMC procedure allows for any degree of hierarchical modelling, it is always possible to check the impact of a given prior by letting its parameters go random. I found that most practitioners are happy with the formalisation of their prior beliefs into mathematical densities, rather than adamant about a specific prior. As for computation, this is not a major issue.

To philosophy…and back

Posted in Books, Statistics, University life with tags , , , , , , on February 16, 2010 by xi'an

Today, I went to listen to Andrew Gelman’s views on the philosophy of Bayesian statistics and this gave me a good opportunity for a 22k bike ride!, as the talk took place in the south-eastern part of the city. (I had not been yet to the new campus of Université Paris Diderot called Paris Rive Gauche. It is brand new, in a renovated district around the Grands Moulins de Paris. The place is buzzing with construction work and the Rue Watt I wanted to visit for its association with Léo Mallet is surrounded by cranes and engines.)

Back to philosophy: Andrew unsurprisingly stated he was not one for conventional philosophical perspectives! He thus went on to demonstrate that Bayesian statistics was not an inductive method but truly an hypothetico-deductive meccanism in the right line of Popper and Lakatos. The main criticism about conventional Bayesian thinking was that Bayesian model choice, by using a discrete collection ot models is inappropriate: on the one hand, models (including priors) can be criticised from the inside. On the other hand, a continuous collective is preferable to the standard model averaging found in Bayesian statistics. Obviously, I do not agree with the ideas that you can test your prior based on the data nor with the fact that the requirement of Bayesian testing on alternatives is a drawback [as we also argued in the Molecular Ecology disputing paper]. But, thanks to all its provocative aspects, this was an enjoyable talk and I think that thru it I understood a bit better Popper’s opposition to induction…

Philosophy of Bayes

Posted in Statistics, University life with tags , , , on February 12, 2010 by xi'an

Again, only for those in Paris next Monday, Andrew Gelman will give a talk at Université Denis Diderot (Paris 7) on Philosophy and the practice of Bayesian statistics in the social sciences at 2pm. It is held in connection with the Institut d’Histoire et de Philosophie des Sciences et des Techniques. I am looking forward to the talk (and to the company of philosophers)!