Archive for frequentist confidence

Philosophies, Puzzles and Paradoxes [book review]

Posted in Books, pictures, Statistics, Travel, University life with tags , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , on May 25, 2024 by xi'an

Yudi Pawitan and Youngjo Lee have written a book that recently caught my attention within the CRC Press list of new publications. Because philosophy, puzzles, and paradoxes are definitely of interest to me (as shown by numerous entries in the ‘Og!). The subtitle of said book is A Statistician’s Search for Truth.

Reviews of the book are already available, with for instance Andrew Gelman stating that he disagrees “with much of this book, but it’s an entertaining and thought-provoking introduction to some challenging questions” or Stephen Senn starting the foreword with “This is a remarkable book: wide-ranging, ambitious, challenging and profound but also intriguing, fascinating and original.” (Senn is also cited within the book for his discussion of our revisit of Harold Jeffreys’ Theory of Probability.) Nice cover as well (albeit I could not trace the origin of it, inside or outside the book.)

The book is made of three parts, one on the philosophical approaches to truth, scientific discovery, deduction, and induction, a second one on probability theories, with philosophical motivations, Bayesian inference, and likelihood-based inference, and a third section on paradoxes. Given that both authors are senior authors who have contributed to likelihood inference throughout their career, incl. the books In All Likelihood and Generalized Linear Models with Random Effects, the likelihood approach is somewhat privileged against other statistical resolutions towards the resolution of the paradoxes, with a defence of confidence distributions and a chapter on epistemic confidence that mostly stems from recent papers by the authors, like Pawitan et al.  (2023) and Lee and Lee (2023). I find the discussion therein somewhat unclear, esp. because the same notation Pr(.) is employed for different probability notions.

“Epistemic confidence is the objective measure of uncertainty that’s attached to single events, where the objectivity is based on a consensus of rational minds.” (p.197)

The philosophy part is following the (European) Enlightenment in producing more and more involved discussions on reason, knowledge and scientific discovery. This exploration is an easy read, as it does not delve particularly deeply in the arguments of Kant, Hume, or Popper. With the apparently unescapable mention of Gödel’s incompleteness theorem, including a sausage citation from Poincaré that reminded of that strip from Tintin in America:which, most probably, he would have applied to Ais! Several sections about pseudo-rational attempts to demonstrate the existence of Dog could have been skipped as well.

The part of probability already considers paradoxes which, like the subsequent ones are mostly the consequence of using natural (and hence ambiguous) languages instead of mathematical descriptions—incl. the statement of the Likelihood Principle. It also discusses Keynes’ logical (or imprecise) probabilities, briefly if appropriately given the pessimistic views of young Keynes on the assessment of the probability of an event. Savage is privileged enough to enjoy an entire chapter discussing his 1950’s axioms leading to the existence of a (subjective)  prior on “the states of the world”. This is followed by a chapter on Inverse probability (aka Bayesian statistics), where the authors consider Bayes’ 1763 Essay to have stayed mostly unnoticed till  the beginning of the 20th Century, which sounds a somewhat subjective judgement. (And as uncovered by Steve Stiegler, the original title of the Essay was indeed intended as a reply to Hume.) A further if short chapter is dedicated to the search for the prior distribution. Which thus gives the misguided impression that there should exist such a thing, rather than acknowledging that Bayesian statements are relative to the prior measure. The remainder of the discussion on invariant and reference priors is however mostly standard. Except when falling for the marginalisation paradox when stating that a product of improper priors implies independence on p.144.

The paradoxes examined in the final part are Allais’ (an alumni of Lycée Lakanal!), and Ellsberg’s, avatars of the Saint Petersburg paradox and referring to failing to adhere to rational decision-making and not in the least to statistics. Conjunction and inclusion “fallacious fallacies”, which are central to Kahneman’s Thinking fast and slow bestseller, and attributed to reasoning in terms of likelihood rather than of probability (without accounting for multiple testing on p.228). A whole if short chapter on the Monty Hall and three prisoners paradoxes, another predictable occurrence in a book on reasoning paradoxes. Again mostly a matter of poor wording, plus relying on the choice of an underlying probability model, for which the authors again follow a likelihood approach, the number of the prize door or of the freed prisoner being the parameter. Kyburg’s (very weak) lottery paradox and related forensic paradoxes, concluding with the rejection of judgements based solely on probability reasoning. Hempel’s paradox of the ravens, a priori unrelated with statistical evidence, but turned into one by squeezing in some sampling models. Finishing with the (envelope) exchange paradox, where the authors refuse to put a prior on the unknown parameter but end up with a solution equivalent to adopting a Jeffreys prior.

In conclusion, this attempt at connecting statistical inference and philosophy, probability concepts and rational decision making, paradoxes and modelling, within a single book is academically sound and overall enjoyable, if not outstanding or remarkable as it does not constitute a radical move away from existing analyses of those classical paradoxes. Furthermore, I find the paradoxes overwhelmingly distant from genuine statistical settings and involving a rather stretched notion of data. Still, methinks I will keep this book in my bookcase, rather than leaving it for the taking in the department coffee room!

As I was completing the book and getting towards writing this book review, I also noticed a two page blurb in Significance (May 2024 issue) written by the authors on their book. (which happens rather frequently with this magazine). Unsurprisingly, the contents provd mostly extracted from the preface and introduction With a nice ravens picture (in conjunction with the raven paradox).

[Disclaimer about potential self-plagiarism: this post or an edited version may eventually appear in my Books Review section in CHANCE.]

Leave a comment »

asymptotics of synthetic likelihood [a reply from the authors]

Posted in Books, Statistics, University life with tags , , , , , , , , , , , , on March 19, 2019 by xi'an

[Here is a reply from David, Chris, and Robert on my earlier comments, highlighting some points I had missed or misunderstood.]

Dear Christian

Thanks for your interest in our synthetic likelihood paper and the thoughtful comments you wrote about it on your blog.  We’d like to respond to the comments to avoid some misconceptions.

Your first claim is that we don’t account for the differing number of simulation draws required for each parameter proposal in ABC and synthetic likelihood.  This doesn’t seem correct, see the discussion below Lemma 4 at the bottom of page 12.  The comparison between methods is on the basis of effective sample size per model simulation.

As you say, in the comparison of ABC and synthetic likelihood, we consider the ABC tolerance \epsilon and the number of simulations per likelihood estimate M in synthetic likelihood as functions of n.  Then for tuning parameter choices that result in the same uncertainty quantification asymptotically (and the same asymptotically as the true posterior given the summary statistic) we can look at the effective sample size per model simulation.  Your objection here seems to be that even though uncertainty quantification is similar for large n, for a finite n the uncertainty quantification may differ.  This is true, but similar arguments can be directed at almost any asymptotic analysis, so this doesn’t seem a serious objection to us at least.  We don’t find it surprising that the strong synthetic likelihood assumptions, when accurate, give you something extra in terms of computational efficiency.

We think mixing up the synthetic likelihood/ABC comparison with the comparison between correctly specified and misspecified covariance in Bayesian synthetic likelihood is a bit unfortunate, since these situations are quite different.  The first involves correct uncertainty quantification asymptotically for both methods.  Only a very committed reader who looked at our paper in detail would understand what you say here.  The question we are asking with the misspecified covariance is the following.  If the usual Bayesian synthetic likelihood analysis is too much for our computational budget, can something still be done to quantify uncertainty?  We think the answer is yes, and with the misspecified covariance we can reduce the computational requirements by an order of magnitude, but with an appropriate cost statistically speaking.  The analyses with misspecified covariance give valid frequentist confidence regions asymptotically, so this may still be useful if it is all that can be done.  The examples as you say show something of the nature of the trade-off involved.

We aren’t quite sure what you mean when you are puzzled about why we can avoid having M to be O(√n).  Note that because of the way the summary statistics satisfy a central limit theorem, elements of the covariance matrix of S are already O(1/n), and so, for example, in estimating μ(θ) as an average of M simulations for S, the elements of the covariance matrix of the estimator of μ(θ) are O(1/(Mn)).  Similar remarks apply to estimation of Σ(θ).  I’m not sure whether that gets to the heart of what you are asking here or not.

In our email discussion you mention the fact that if M increases with n, then the computational burden of a single likelihood approximation and hence generating a single parameter sample also increases with n.  This is true, but unavoidable if you want exact uncertainty quantification asymptotically, and M can be allowed to increase with n at any rate.  With a fixed M there will be some approximation error, which is often small in practice.  The situation with vanilla ABC methods will be even worse, in terms of the number of proposals required to generate a single accepted sample, in the case where exact uncertainty quantification is desired asymptotically.  As shown in Li and Fearnhead (2018), if regression adjustment is used with ABC and you can find a good proposal in their sense, one can avoid this.  For vanilla ABC, if the focus is on point estimation and exact uncertainty quantification is not required, the situation is better.  Of course as you show in your nice ABC paper for misspecified models jointly with David Frazier and Juidth Rousseau recently the choice of whether to use regression adjustment can be subtle in the case of misspecification.

In our previous paper Price, Drovandi, Lee and Nott (2018) (which you also reviewed on this blog) we observed that if the summary statistics are exactly normal, then you can sample from the summary statistic posterior exactly with finite M in the synthetic likelihood by using pseudo-marginal ideas together with an unbiased estimate of a normal density due to Ghurye and Olkin (1962).  When S satisfies a central limit theorem so that S is increasingly close to normal as n gets large, we conjecture that it is possible to get exact uncertainty quantification asymptotically with fixed M if we use the Ghurye and Olkin estimator, but we have no proof of that yet (if it is true at all).

Thanks again for being interested enough in the paper to comment, much appreciated.

David, Chris, Robert.

Leave a comment »