Archive for p-values

on stopping rules

Posted in Books, Statistics with tags , , , , , , , , , , , , , , , , , on August 3, 2025 by xi'an

The workshop in Chennai and its focus on sequential procedures made me realise (among other things) I had never read Cornfield’s 1966 TAS paper on sequential testing and the likelihood principle:

“By sequential analysis I mean any form of analysis in which the conclusion depends not only on the data, but also on the stopping rule.”

Written with little maths and formalism, this paper argues that keeping a fixed critical level amounts to keeping a fixed amount of evidence. Hence constituting an early critique of p-values even though not expressed in such terms. The part of the paper related with the likelihood principle does not address testing or evidence in a Bayesian way. As a side (late awakening) remark, iid observations in sequential settings are not longer independent, conditional on the stopping rule realisation N=n, since they are constrained by the fact that the stopping rule realisation is n and not n-1, n-2, …  For a short while, I thought it was in turn impacting the distribution of any “sufficient” statistic one may propose, with a normalising constant that depends on the unknown parameter and hence cannot be neglected. Over all those years, I had never though of the modification of sufficiency characteristics in such contexts. But in fine the pair made of the value of the stopping rule and of the unsequential sufficient statistics proves enough. And the normalisation constant is the probability that the stopping rule.. stops!, which is equal to one! For the same short while, I was then wondering that the stopping rule principle!

“my second line of argument that there is a reasonable alternative explication of the idea of inference and one which leads to the rejection of sequential analysis. This explication is provided by the likelihood principle—which states that all observations leading to the same likelihood function should lead to the same conclusion.”

I thus went back to the fundamentals (!), namely [freely available] Bernardo’s and Smith’s Section 5.1.4 (reproduced in EJ’s Stopping rule appendix, also citing Cornfield at length), where the likelihood is properly defined by the joint density of the stopping rule τ and the attached sample at their realised values. And failing in the end (and a discussion with Judith)nto spot a missing normalisation constant.

e-values in Chennai

Posted in Books, pictures, Running, Statistics, Travel, University life with tags , , , , , , , , , , , , , , , , , , , on July 23, 2025 by xi'an

To recap, I thus attended the BIRS-CMI workshop 25w5482 at the Chennai Mathematical Institute, Navalur, Tamil Nadu, in early July, for being intrigued by the developments around the concept. And enjoyed the week, from partaking in the company of friendly and enthusiastic academics to the exposure of new views and concepts, mostly remote from mine’s. Recall that an e-value attached to an hypothesis H described as a collection of distributions is a non-negative random variable E with expectation less than 1 for E~Q and all Q ∈ H. When a stopping rule is involved, the e-value is extended into an e-process. (Beyond Aaditya Ramdas’ E-book, Ruodu Wang also wrote a “tiny” review.) Aaditya Ramdas recalled in his introduction of the workshop that e-values are fundamentally equivalent to p-values and confidence intervals. And that a confidence sequence is a sequence of confidence intervals that contains the true value for all time steps t’s with a probability of at least 1-α.

The talks reflected a general belief in α levels and in Neyman-Pearsonian likelihood ratio optimality in simple vs simple settings, considering extension for sequential analysis settings, anytime inference, universality under general alternatives, and connections with FDRs, incl. Benjamini & Hochberg solution, but pointed out a lack of middle ground between frequentists and Bayesians.

“e-values have a clear interpretation in terms of betting and are closely related to likelihood ratios and other Bayes factor. At the same time, e–values do not require prior distributions conditional on the null and alternative hypotheses”

Although David R. Bickel attempted a Bayesian version, using a marginal likelihood ratio within betting settings, that is an incoming American Statistician paper. I may have being missing some aspects due to a lack of sleep the night before (!), but I find the attempt resulting in a fairly unusual vision of Bayesian testing as either not depending on any parameter or on the opposite using a family of priors. I did not understand either the “criticism” that the predictive depends on the prior and felt that this representation was bending in a rather onsiderable way the Bayesian perspective towards achieving a certain degree of agreement with p– and e-value notions, to conclude that the Bayes factor is an e-value. (As an aside, this may be the first paper that cited our critical review of Aitkin! Similarly, Shubhada Agrawal mentioned Roger Farrell in his talk, with whom we wrote a complete class Annals paper in the late 1980’s.) Nikos Ignatiadis also explored Empirical Bayes e-values, while Ben Chugg gave a presentation (constrained) admissibility, albeit under type-I error constraints that makes Bayes infeasible and using Neyman-Pearsonian loss functions. On the last day, Peter Grünwald tried for some BFF cohesion with openings on e-posteriors, treating hypothesis testing losses symmetrically, defining it as an inverse of e-values but incorporating pseudo-posteriors of many flavours like confidence, inferential, and fiducial distributions. He also mentioned a Savage-Dickey version while using an arbitrary prior, which is also an e-value, but with upper & lower meanings, again with measure issues

Given the hosting of the workshop in the Chennai Mathematical Institute, which is quite far from the centre of town (much closer to Mahabalipuram!), I did not visit Chennai but enjoyed the South Indian cuisine (albeit missing some fierceness in the spices!) and local fruits from street stands, if being sorry I could not find cocoa pods from nearby Kerala.

A modern introduction to probability and statistics [book review]

Posted in Books, R, Statistics, Travel, University life with tags , , , , , , , , , , , , , , , , , , , , , on July 12, 2025 by xi'an

In the plane to Bengaluru, I read through the book A modern introduction to probability and statistics, by Graham Upton—whose Measuring Animal Abundance I reviewed for CHANCE a while ago—, which is based on the earlier Understanding Statistics, written jointly with Ian Cook. (Not to be confused with A modern introduction to probability and statistics by Dekking et al.) The subtitle is understanding statistical principles in the computer age. Sorry, in the age of the computer. While the cover is most pleasant (and modern), as noticed by an AF flight attendant, the contents are very very standard and could have been written decades ago since the main concession to “the” computer age is the inclusion of a few R commands at the end of most chapters. There are even a few distribution tables here and there (in case “the” computer is not available). But there is no other connection with computational statistics or statistical computing.

The classicism of the contents and the intended audience mean there is little therein on which to either object or criticise. The mixture of elementary probability and basic statistics in a single textbook always feels awkward to me and I think I would have trouble teaching solely from this material. Apart from the glaring typo on the variance of the sum of two correlated random variables on page 87, missing the factor 2 in front of the covariance, while correct(ed) p97 (and the inevitable “the the” typo spotted once). My main criticisms are on the potential confusion between samples and populations in the early chapters, when some statistics are used as motivational examples, as for instance in a (hidden) Monte Carlo stabilisation to the limiting values (p57), way before the Law of Large Numbers is introduced,, the variable mileage in mathematical rigour (while being uncertain that first year students can handle integrals and derivatives), the textbook examples, and the amount of the book contents spent on descriptive statistics and even more on the “classical” tests, with no critical perspective on using point nulls or p-values. The book concludes with a four page (benevolent) chapter on Bayesian statistics that is superfluous imho, or even counterproductive since my experience with a rushed introduction to Bayesian principles almost always result in a rejection of said principles. Plus, the illustration with the coin tossing is not particularly helpful since Andrew maintains that one can load a die, but cannot bias a coin. (A similar reservation on the half-page 289 coverage on pseudo-random generation and Monte Carlo principles for computing p-values.)

Minor (mostly idiosyncratic) remarks follow: CLT prior to LLN,   n-1 in sample sd, little to no model criticism (ntbcf goodness of fit), missing an opportunity when mentioning the varying probability of a day being a birthday (p31) in contrast with BDA cover story, and another opportunity to cite the 2024 Ig Nobel Prize for coin tossing around the LLN, an unclear definition for random variables( p53) and a potentially confusing introduction of Poisson distributions through a informal reference to Poisson processes (and no reason why the years of accession of the kings of Sussex and England till Guillaume—making a return on p178 with the Domesday Book—in 1066 should follow such a process as suggested in Figure 3.5), a surprising definition of the constant e as the special case of exp(x) when x=1 and its series expansion (p70), omitting proofs on laws of sums of iid rv’s by introducing moment generating functions rather late, another obscure reference to a 16th German treatise on surveying as a precursor of the CLT (p131), a proof for the normalising constant of the Normal density that will most likely escape most first year students, a introduction of the t, F, and χ² distributions with no mention of their respective densities (pp141-147), never defining a joint Normal distribution density, insisting on unbiasedness without noting that maximum likelihood—with a strange motivation that it “makes the next sample of n observations most likely to resemble the data in the current sample (p228)—estimators are almost always biased, an abundance of footnotes that may prove of little interest for the youngest readers.

[Disclaimer about potential self-plagiarism as usual: this post or an edited version will eventually appear in my Books Review section in CHANCE.]

off to Chennai

Posted in Statistics, Travel, University life with tags , , , , , , , , , , , , , , , , on June 29, 2025 by xi'an

Today, I am off to Chennai (aka Madras) for a… Banff workshop! As one of its several branches and associated institutions (incl. Oaxaca), BIRS has the Chennai Mathematical Institute hosting the workshop “Game-theoretic statistical inference: Optional Sampling, Universal Inference, and Multiple Testing based on e-values”, which I attend in an attempt to better ascertain e-values. (Obviously, George Pérec could not have been invited!)

Nice meeting!

Posted in pictures, R, Running, Statistics, Travel, University life with tags , , , , , , , , , , , , , , , , , , , , , , , on December 18, 2024 by xi'an

The ICSDS 2024 meeting in Nice is quite impressive and not primarily because it is in Nice under a beautiful December sun. As other (numerous) IMS meetings I attended (since the initial one in Uppsala in 1990!), the program is of high quality and along topics that are currently moving fast or emerging. From the sessions I attended, e-values are strongly represented, although it remains unclear to me why they should constitute a major departure from p-values, as they stick to hypothesis testing, Type I error, power, and the whole paraphernalia of Neyman-Pearson formalism. If I manage to attend a BIRS workshop on the subject next Summer, I may manage to get a better e-derstanding!The MCMC (only!) session included a presentation by Guanyang Wang that generalised different approximate MCMC schemes into a unified one. And one by Filippo Ascolani on Gibbs beating the competition! I also attended the Bayesian prediction session, where my friends Sonia Petrone and Chris Holmes have presentations on their respective Series B papers. I discussed both on the ‘Og, on 15 March 2023 and 07 November 2022, respectively. This time, I found that both talks had a Bayesian bootstrap flavour, which is not surprising when considering the non-parametric nature of the approach. And they left me wondering at it being protected from overfitting.
My only plenary session was Cynthia Dwork’s on outcome indistinguishability, which, while related to the privacy topics I was topic, remained somewhat obscure as to its purpose. Meaning I have to get through the paper to get a more holistic perspective.
Of course, Nice in Winter is a very nice place, with the waterfront available for running an uninterrupted 15km as we found out with Jérémie Houssineau (at a brisk 4’09” pace I had not planned before starting!) and the sea all for myself (for a dozen minutes before losing digits!). Unfortunately I had to skip the final day due to examinations of the Paris Dauphine MASH master. And miss Stan receiving a student award. But I am looking forward the next iterations of ICSDS. (Not including Copenhagen, Madrid and many many other places in 2025, since ICSDS seemed a most common name for conferences, some presumably predatory! The true location is Sevilla, to keep up with the Mediterranean theme of ICSDS!)