I spotted this title in the new arXiv postings on Monday. When Is Generalized Bayes Bayesian? A Decision-Theoretic Characterization of Loss-Based Updating by Kenichiro McAlinn & Kōsaku Takanashi is discussing decision-theoretic consequences of generalized Bayes approaches based on losses and show that decisions based on a loss-based posterior coincides with those of ordinary Bayes if and only if the loss is essentially a negative log-likelihood (leading to a belief posterior). This is not very surprising in that, otherwise, there is no Bayesian update delivering the generalised Bayes pseudo-posteriors (which can be traced back to a 2007 result of Catoni). The authors also demonstrate that generalized marginal likelihoods are not delivering evidence for decision posteriors, and thus that Bayes factors are not well-defined in this context, which reminds me of our warning for ABC model choice. However, the reason here is much more mundane, as it is due to the decision posterior failing to identify the normalising constant Z(x). Outside belief posteriors. The paper concludes with a coherence book, which is a table reproduced above.
Archive for loss functions
When Is Generalized Bayes Bayesian?
Posted in Books, Statistics, University life with tags ABC, Bayes factors, Bayesian model choice, coherence, decision theory, generalised Bayesian inference, loss functions, marginal likelihood, normalising constant on February 13, 2026 by xi'anNature tidbits
Posted in Books, pictures, Statistics, University life with tags AI, Brave New World, climate change, Clockwork Orange, CO2 levels, cover, frogs, Huber loss, Intergovernmental Panel on Climate Change, loss functions, machine learning, Nature, neural networks and learning machines, neuroscience of religions, sauna, temperature anomalies, ZRR on September 2, 2024 by xi'an
- a “technology and tools” long article on picking the “right” loss function when constructing an algorithmic predictor or another ML tool,
although the author remains vague about the “rightness” part (with an incorrect entry for Huber loss). The message is however clearly that tuning the loss function to the problem one wants to address is (obviously) key. With additional advices about properly handling outliers, avoiding overfitting, and correctly modelling noise. In short, run a proper statistical analysis! - a call for neuroscientists not to be afraid of studying religions, which should sound like an obviousness, but believers and religious leaders are often so touchy about their faith that running representative scientific studies of the “brain processes associated with religiosity and spirituality” may prove inaccessible. I also find the statement “researchers might be able to get a better handle on what (if any) alterations happen in people’s brains in the rare instances when religious belief turns to radicalized action or sectarian hatred” closer to Brave New World or Clockwork Orange than to the purpose of Nature!
- the warmest summer ever, ever, ever…
- a scary story from South Korea where an academic got sentenced to two years in prison for sharing data with Chinese collaborators on “national core technology”, a case reminiscent of similar ones earlier in the US. And where the academic researcher is again seen as the sole culprit instead of their university or the government sharing the responsibility. In France, the imminent threat to turn mathematics and statistics labs into restrictive regime zones (ZRR), with much heavier constraints on visitors and students, and on publication contents, carried by the researchers themselves, pertains from the same tendency. (A personal illustration of the induced administrative absurdities: my datascience lab being next to a biomedical ZRR lab, I am not allowed to use the lift shared by both units, but can nonetheless reach the same spot using the nearby stairs.)
O’Bayes 19/1 [snapshots]
Posted in Books, pictures, Statistics, University life with tags ABC algorithm, Bayesian decision theory, clustering, Galaxy, imprecise probabilities, indoor swimming, Jeffreys-Lindley paradox, loss functions, maximum entropy, mixtures of distributions, O'Bayes 2019, objective Bayes, PC priors, prior assessment, score function, Society for Imprecise Probability, University of Warwick, Valencia meeting on June 30, 2019 by xi'anAlthough the tutorials of O’Bayes 2019 of yesterday were poorly attended, albeit them being great entries into objective Bayesian model choice, recent advances in MCMC methodology, and the multiple layers of BART, for which I have to blame myself for sticking the beginning of O’Bayes too closely to the end of BNP as only the most dedicated could achieve the commuting from Oxford to Coventry to reach Warwick in time, the first day of talks were well attended, despite weekend commitments, conference fatigue, and perfect summer weather! Here are some snapshots from my bench (and apologies for not covering better the more theoretical talks I had trouble to follow, due to an early and intense morning swimming lesson! Like Steve Walker’s utility based derivation of priors that generalise maximum entropy priors. But being entirely independent from the model does not sound to me like such a desirable feature… And Natalia Bochkina’s Bernstein-von Mises theorem for a location scale semi-parametric model, including a clever construct of a mixture of two Dirichlet priors to achieve proper convergence.)
Jim Berger started the day with a talk on imprecise probabilities, involving the society for imprecise probability, which I discovered while reading Keynes’ book, with a neat resolution of the Jeffreys-Lindley paradox, when re-expressing the null as an imprecise null, with the posterior of the null no longer converging to one, with a limit depending on the prior modelling, if involving a prior on the bias as well, with Chris discussing the talk and mentioning a recent work with Edwin Fong on reinterpreting marginal likelihood as exhaustive X validation, summing over all possible subsets of the data [using log marginal predictive].
Håvard Rue did a follow-up talk from his Valencià O’Bayes 2015 talk on PC-priors. With a pretty hilarious introduction on his difficulties with constructing priors and counseling students about their Bayesian modelling. With a list of principles and desiderata to define a reference prior. However, I somewhat disagree with his argument that the Kullback-Leibler distance from the simpler (base) model cannot be scaled, as it is essentially a log-likelihood. And it feels like multivariate parameters need some sort of separability to define distance(s) to the base model since the distance somewhat summarises the whole departure from the simpler model. (Håvard also joined my achievement of putting an ostrich in a slide!) In his discussion, Robin Ryder made a very pragmatic recap on the difficulties with constructing priors. And pointing out a natural link with ABC (which brings us back to Don Rubin’s motivation for introducing the algorithm as a formal thought experiment).
Sara Wade gave the final talk on the day about her work on Bayesian cluster analysis. Which discussion in Bayesian Analysis I alas missed. Cluster estimation, as mentioned frequently on this blog, is a rather frustrating challenge despite the simple formulation of the problem. (And I will not mention Larry’s tequila analogy!) The current approach is based on loss functions directly addressing the clustering aspect, integrating out the parameters. Which produces the interesting notion of neighbourhoods of partitions and hence credible balls in the space of partitions. It still remains unclear to me that cluster estimation is at all achievable, since the partition space explodes with the sample size and hence makes the most probable cluster more and more unlikely in that space. Somewhat paradoxically, the paper concludes that estimating the cluster produces a more reliable estimator on the number of clusters than looking at the marginal distribution on this number. In her discussion, Clara Grazian also pointed the ambivalent use of clustering, where the intended meaning somehow diverges from the meaning induced by the mixture model.
the philosophical importance of Stein’s paradox [a reply from the authors]
Posted in Books, pictures, Statistics, University life with tags Bayesian Analysis, Bayesian Choice, Charles Stein, decision theory, frequentist inference, James-Stein estimator, loss functions, philosophy of sciences, Stein effect, Stein's phenomenon, Stephen Stigler on January 15, 2016 by xi'an
[In the wake of my comment on this paper written by three philosophers of Science, I received this reply from Olav Vassend.]
Thank you for reading our paper and discussing it on your blog! Our purpose with the paper was to give an introduction to Stein’s phenomenon for a philosophical audience; it was not meant to — and probably will not — offer a new and interesting perspective for a statistician who is already very familiar with Stein’s phenomenon and its extensive literature.
I have a few more specific comments:
1. We don’t rechristen Stein’s phenomenon as “holistic pragmatism.” Rather, holistic pragmatism is the attitude to frequentist estimation that we think is underwritten by Stein’s phenomenon. Since MLE is sometimes admissible and sometimes not, depending on the number of parameters estimated, the researcher has to take into account his or her goals (whether total accuracy or individual-parameter accuracy is more important) when picking an estimator. To a statistician, this might sound obvious, but to philosophers it’s a pretty radical idea.
2. “The part connecting Stein with Bayes again starts on the wrong foot, since it is untrue that any shrinkage estimator can be expressed as a Bayes posterior mean. This is not even true for the original James-Stein estimator, i.e., it is not a Bayes estimator and cannot be a Bayes posterior mean.”
That seems to depend on what you mean by a “Bayes estimator.” It is possible to have an empirical Bayes prior (constructed from the sample) whose posterior mean is identical to the original James-Stein estimator. But if you don’t count empirical Bayes priors as Bayesian, then you are right.
3. “And to state that improper priors “integrate to a number larger than 1” and that “it’s not possible to be more than 100% confident in anything”… And to confuse the Likelihood Principle with the prohibition of data dependent priors. And to consider that the MLE and any shrinkage estimator have the same expected utility under a flat prior (since, if they had, there would be no Bayes estimator!).”
I’m not sure I completely understand your criticisms here. First, as for the relation between the LP and data-dependent priors — it does seem to me that the LP precludes the use of data-dependent priors. If you use data from an experiment to construct your prior, then — contrary to the LP — it will not be true that all the information provided by the experiment regarding which parameter is true is contained in the likelihood function, since some of the information provided by the experiment will also be in your prior.
Second, as to our claim that the ML estimator has the same expected utility (under the flat prior) as a shrinkage prior that it is dominated by—we incorporated this claim into our paper because it was an objection made by a statistician who read and commented on our paper. Are you saying the claim is false? If so, we would certainly like to know so that we can revise the paper to make it more accurate.
4. I was aware of Rubin’s idea that priors and utility functions (supposedly) are non-separable, but I didn’t (and don’t) quite see the relevance of that idea to Stein estimation.
5. “Similarly, very little of substance can be found about empirical Bayes estimation and its philosophical foundations.”
What we say about empirical Bayes priors is that they cannot be interpreted as degrees of belief; they are just tools. It will be surprising to many philosophers that priors are sometimes used in such an instrumentalist fashion in statistics.
6. The reason why we made a comparison between Stein estimation and AIC was two-fold: (a) for sociological reasons, philosophers are much more familiar with model selection than they are with, say, the LASSO or other regularized regression methods. (b) To us, it’s precisely because model selection and estimation are such different enterprises that it’s interesting that they have such a deep connection: despite being very different, AIC and shrinkage both rely on a bias-variance trade-off.
7. “I also object to the envisioned possibility of a shrinkage estimator that would improve every component of the MLE (in a uniform sense) as it contradicts the admissibility of the single component MLE!”
I don’t think our suggestion here contradicts the admissibility of single component MLE. The idea is just that if we have data D and D’ about parameters φ and φ’, then the estimates of both φ and φ’ can sometimes be improved if the estimation problems are lumped together and a shrinkage estimator is used. This doesn’t contradict the admissibility of MLE, because MLE is still admissible on each of the data sets for each of the parameters.
Again, thanks for reading the paper and for the feedback—we really do want to make sure our paper is accurate, so your feedback is much appreciated. Lastly, I apologize for the length of this comment.
Olav Vassend
the philosophical importance of Stein’s paradox
Posted in Books, pictures, Statistics, University life with tags Bayesian Analysis, Bayesian Choice, Charles Stein, decision theory, frequentist inference, James-Stein estimator, loss functions, philosophy of sciences, Stein effect, Stein's phenomenon, Stephen Stigler on November 30, 2015 by xi'anI recently came across this paper written by three philosophers of Science, attempting to set the Stein paradox in a philosophical light. Given my past involvement, I was obviously interested about which new perspective could be proposed, close to sixty years after Stein (1956). Paper that we should actually celebrate next year! However, when reading the document, I did not find a significantly innovative approach to the phenomenon…
The paper does not start in the best possible light since it seems to justify the use of a sample mean through maximum likelihood estimation, which only is the case for a limited number of probability distributions (including the Normal distribution, which may be an implicit assumption). For instance, when the data is Student’s t, the MLE is not the sample mean, no matter how shocking that might sounds! (And while this is a minor issue, results about the Stein effect taking place in non-normal settings appear much earlier than 1998. And earlier than in my dissertation. See, e.g., Berger and Bock (1975). Or in Brandwein and Strawderman (1978).)
While the linear regression explanation for the Stein effect is already exposed in Steve Stigler’s Neyman Lecture, I still have difficulties with the argument in that for instance we do not know the value of the parameter, which makes the regression and the inverse regression of parameter means over Gaussian observations mere concepts and nothing practical. (Except for the interesting result that two observations make both regressions coincide.) And it does not seem at all intuitive (to me) that imposing a constraint should improve the efficiency of a maximisation program… Continue reading
