Archive for model choice

« Older Entries
Newer Entries »

Deviance and posterior likelihood

Posted in Statistics with tags , , , on July 5, 2009 by xi'an

An interesting paper by Murray Aitkin, Charles Liu, and Tom Chadwick is due to appear in the Annals of Applied Statistics. Its main focus is the comparison of models for a specific problem of small area estimation, but it also contains general views on Bayesian model choice that relates to our earlier paper about DIC and to our reassessment of Bayes factor approximation with Jean-Michel Marin. In connection with Aitkin’s 1991 paper, this paper considers the posterior distribution of the likelihood. For one given model, this amounts to using the likelihood twice since the posterior expectation of the likelihood equals

\int_{\Theta} \pi(\theta|x) L(\theta|x) \text{d}\theta = \int_{\Theta} \pi(\theta) L^2(\theta|x) \text{d}\theta / m(x)

This made me first think that DIC may suffer from the same drawback since

\text{DIC} = -4 \mathbb{E}[\log L(\theta|x) | x] + 2 \log L(\hat\theta(x)|x)

hence integrates a quantity that depends on x against the posterior, but this is not the same since

\mathbb{E}[\log L(\theta|x) | x] = \int_{\Theta} \log(\pi(\theta|x)/\pi(\theta)) \pi(\theta|x)\text{d}\theta - \log(m(x))

appears like a Kullback-Leibler divergence between the prior and the posterior distributions. Back to the paper, it suggest to run model comparison via the distribution of the likelihood ratio values

\dfrac{Lj(\theta_i|x)}{Lk(\theta_k|x)}

where the \theta_i‘s and \theta_k‘s are draws from the respective posteriors. This seems very close to Steve Scott’s (JASA, 2002) and to Peter Congdon’s (CSDA, 2006) solutions analysed in our paper, in that MC(MC) runs are ran for each model separately and the samples are gathered together to produce either the posterior expectation (in Scott’s case) or the posterior distribution (for the current paper) of

\rho_i L(\theta_i|x) \bigg/ \sum_k \rho_i L(\theta_i|x)

which do not correspond to genuine Bayesian solutions, in my opinion, because the data x is used repeatedly in this process, producing a sample from the product of the posteriors, instead of using pseudo-priors as in Carlin and Chib (Series B, 1995). In addition, I fail to see how to interpret the “posterior” distribution of this quantity, even under the condition of using pseudo-priors, as it will also reflect the joint distribution, while a coherent model choice procedure should only depend on the marginal (model-wise) distributions (which reminds me of an old criticism against Pitman nearness…)

6 Comments »

ABC for model criticism

Posted in Statistics with tags , , , , on June 16, 2009 by xi'an

An ABC paper by Oliver Ratmann, Christophe Andrieu, Carsten Wiuf and Sylvia Richardson has just appeared on PNAS Early Edition, (It will also be presented at the “ABC in Paris” meeting of June 26.) It is about the use of the ABC approximation error \epsilon in an altogether different way, namely as a tool assessing the goodness of fit of a given model. It reminds me of Richard Wilkinson’s earlier ABC paper, discussed on that post, but the scope is somehow different. The fundamental idea is to use \epsilon as an additional parameter of the model, simulating from a joint posterior distribution

f(\theta,\epsilon|x_0) \propto \xi(\epsilon|x_0,\theta)\times\pi_\theta(\theta)\times\pi_\epsilon(\epsilon)

where x_0 is the data and \xi(\epsilon|x_0,\theta) plays the role of the likelihood. (The \pi's are obviously the priors on \theta and \epsilon.) In fact, \xi(\epsilon|x_0,\theta) is the prior predictive density of \rho(S(x),S(x_0)) given \theta and x_0 when x is distributed from f(x|\theta). The authors then derive an ABC algorithm they call ABCμ to simulate an MCMC chain targeting this joint distribution, replacing \xi(\epsilon|x_0,\theta) with a non-parametric kernel approximation. For each model under comparison, the marginal posterior distribution on the error \epsilon is then used to assess the fit of the model, the logic of it being that this posterior should include 0 in a reasonable credible interval. (Contrary to other ABC papers, \epsilon can be negative and multidimensional in this paper.)

As written above, this is a very interesting paper, full of innovations, that should span new directions in the way one perceives ABC. It is also quite challenging, partly due to the frustrating constraints PNAS imposes on the organisation (and submission) of papers. The paper thus contains a rather sketchy main part, a Materials and Methods addendum, and a Supplementary Material file! Flipping back and forth between those files certainly does not improve reading. I have never understood why PNAS was is so rigid about a format that does not suit non-experimental sciences…

Given the wealth of innovations contained in the paper, I will certainly post again on it, but let me add here that, while the authors stress they use the data once (a point always uncertain to me), they also define the above target by using simultaneously a prior distribution on \epsilon and a conditional distribution on the same \epsilon—that they interpret as the likelihood in (\epsilon,\theta). The product being most often defined as a density in (\epsilon,\theta), it can be simulated from, but I have trouble seeing this as a regular Bayesian problem, especially because it seems the prior on \epsilon significantly contributes to the final assessment (but is not particularly discussed in the paper, except in the S1.10 section).

Another Bayesian conundrum is the fact that both \theta and \epsilon are taken to be the sames across models. In a sense, I presume \theta can be completely different, but using the same prior on \epsilon over all models under comparison is more of an issue…

8 Comments »

3rd Rimini Bayesian workshop

Posted in Statistics, University life with tags , , , , on June 1, 2009 by xi'an

I have (recklessly) agreed to take part in a debate at the third Rimini Bayesian econometrics workshop, taking place at the Rimini Center for Economic Analysis next July 1 and 2. The theme of the debate is “The 21st Century Belongs to Bayes”, which sounds a wee too religious to my taste!, and I will argue about this with Russell Davison, both from McGill and GREQAM in Marseilles. I hope this won’t turn into a disaster as I am not particularly gifted at debating and even less at preaching. Here are the preliminary slides I drafted yesterday

recycling slides from my model choice talk as well as from my master course on The Bayesian Choice. The main input at this stage is the insertion of quotes from recent papers by Andrew Gelman and Alan Templeton, as well as from the discussions of the former by José Bernardo, Stephen Senn, and Larry Wasserman. The quotes from Andrew may sound apocryphal but they come from his April Fool 2008 post (and then Bayesian Analysis paper) where he covers some of the anti-Bayesian arguments. My reasoning in pilling in all those quotes is to douse the most standard criticisms from the start, with a tongue-in-cheek attitude since I completely agree with the first quote from Andrew. However, this most likely means that Russell will look elsewhere for arguments against a 21st Bayesian century. I can actually predict with a 87% prior probability that one such argument will be about large dimensional models, which is another standard criticism addressed at the Bayesian methodology.

1 Comment »

Bayesian p-values (2)

Posted in Statistics with tags , , on May 8, 2009 by xi'an

“What the use of P implies, therefore, is that a hypothesis that may be true may be rejected because it has not predicted observable results that have not occurred.” H. Jeffreys, Theory of Probability

Looking a bit further into the literature about Bayesian p-values, I read the Statistical Science paper of Bayarri and Castellanos on Bayesian Checking of the Second Levels of Hierarchical Models, which considers extensions of the surprise measures of Bayarri and Berger (2000, JASA, see preprint) in the setting of a normal hierarchical model. (Here is a link to a very preliminary version of the paper.) While I quite appreciate the advances contained in those papers, I am still rather reticent to put forward those measures of surprise as decision tools and, given the propensity of users to hijack tools towards their own use, I fear they would end up being used exactly as p-values, namely interpreted as probabilities of the null hypothesis.

Here are some comments on the paper that I made during my trip back from Montpellier yesterday evening. The most natural tool in building a Bayesian p-value seems to me to use the probability of a tail event under the predictive

\mathbf{h(y|x) = \int f(y|\theta) \pi(\theta|x) \text{d} \theta}

and using P^h(t(Y)\ge t(x) |x) to define the “surprise” means that

  1. the tail event is evaluated under a distribution that is “most” favourable to x, since it is based on the posterior distribution of \theta given x. [This point somehow relates to the “using the data twice” argument that I do not really understand in this setting: conditional on x, this is a proper Bayesian way of evaluating the probability of the event \{t(Y)\ge t(x)\}. (What one does with this evaluation is another issue!)]

  2. following Andrew Gelman’s discussion, there is no accounting for the fact that “all models are wrong” and that we are working from within a model trying to judge of the adequacy of this model, in a Munchausen’s way of pull oneself up to the Moon. Again, there is a fair danger in using P^h(t(Y)\ge t(x) |x) as the posterior probability of the model being “true”…

  3. I think the approach does not account for the (decisional) uses of the numerical evaluation, hence is lacking calibration: Is a value of 10¯² small?! Is a value of 10¯³ very small?! Are those absolute or relative values?! And if the value is used to decide for or against a model, what are the consequences of this decision?

  4. the choice of the summary statistic t(x) is quite relevant for the value of the surprise measure and there is no intrinsic choice. For instance, I first thought using the marginal likelihood m(x) would be a relevant choice, but alas this is not invariant under a change of variables.

  5. another [connected] point that is often neglected in model comparison and model evaluation is that sufficient statistics are only sufficient within a given model, but not for comparing nor evaluating this model. For instance, when comparing a Poisson model with a negative binomial model, the sum of the observation is sufficient in both cases but not in the comparison!

2 Comments »

Great week!

Posted in Statistics with tags , , , on April 7, 2009 by xi'an

Within a week, I got acceptance letters from Bayesian Analysis, Biometrika, and Statistical Science for the papers ABC likelihood-free methods for model choice in Gibbs random fields (arXiv posting), ABC-PMC (arXiv posting), and Theory of Probability revisited (arXiv posting) What a great week! Further, the paper on Jeffreys will be a discussion paper with two discussions from Andrew Gelman and Dennis Lindley already received. The ABC-PMC paper, now called Adaptive approximate Bayesian computation, is in the Miscellanea section of Biometrika, which I feel is a bit lower key, but, nonetheless, having this re-evaluation of ABC-PRC and of the substitute ABC-PMC we propose in a top-tier journal is the major thing.

1 Comment »

« Older Entries
Newer Entries »