Here are the slides of my tutorial at O’ Bayes 2013 today, a pot-pourri of various, recent and less recent, criticisms (with, albeit less than usual, a certain proportion of recycled slides):
[speakerdeck https://speakerdeck.com/xianblog/obayes-2013-duke-university-a-tutorial-on-alternative-bayesian-tests%5D
“The difficulty in constructing a Bayesian hypothesis test arises from the requirement to specify an alternative hypothesis.”
Vale Johnson published (and arXived) a paper in the Annals of Statistics on uniformly most powerful Bayesian tests. This is in line with earlier writings of Vale on the topic and good quality mathematical statistics, but I cannot really buy the arguments contained in the paper as being compatible with (my view of) Bayesian tests. A “uniformly most powerful Bayesian test” (acronymed as UMBT) is defined as
“UMPBTs provide a new form of default, nonsubjective Bayesian tests in which the alternative hypothesis is determined so as to maximize the probability that a Bayes factor exceeds a specified threshold”
which means selecting the prior under the alternative so that the frequentist probability of the Bayes factor exceeding the threshold is maximal for all values of the parameter. This does not sound very Bayesian to me indeed, due to this averaging over all possible values of the observations x and comparing the probabilities for all values of the parameter θ rather than integrating against a prior or posterior and selecting the prior under the alternative with the sole purpose of favouring the alternative, meaning its further use when the null is rejected is not considered at all and catering to non-Bayesian theories, i.e. trying to sell Bayesian tools as supplementing p-values and arguing the method is objective because the solution satisfies a frequentist coverage (at best, this maximisation of the rejection probability reminds me of minimaxity, except there is no clear and generic notion of minimaxity in hypothesis testing).
 | David on Bayesian workflow [book review… |
 | Ewan on Tidy up, Claude! |
 | xi'an on Tidy up, Claude! |
| Ewan on Tidy up, Claude! |
| Ewan on Já við viðræðum. Já við Evrópu… |
Valen in Le Monde
Posted in Books, Statistics, University life with tags blogging, comments, False positive, Le Monde, Monsanto, p-values, Passeur de Sciences, statistical significance, UMPB test, uniformly most powerful tests, Valen Johnson on November 21, 2013 by xi'anValen Johnson made the headline in Le Monde, last week. (More precisely, to the scientific blog Passeur de Sciences. Thanks, Julien, for the pointer!) With the alarming title of “Une étude ébranle un pan de la méthode scientifique” (A study questions one major tool of the scientific approach). The reason for this French fame is Valen’s recent paper in PNAS, Revised standards for statistical evidence, where he puts forward his uniformly most powerful Bayesian tests (recently discussed on the ‘Og) to argue against the standard 0.05 significance level and in favour of “the 0.005 or 0.001 level of significance.”
While I do plan to discuss the PNAS paper later (and possibly write a comment letter to PNAS with Andrew), I find interesting the way it made the headlines within days of its (early edition) publication: the argument suggesting to replace .05 with .001 to increase the proportion of reproducible studies is both simple and convincing for a scientific journalist. If only the issue with p-values and statistical testing could be that simple… For instance, the above quote from Valen is reproduced as “an [alternative] hypothesis that stands right below the significance level has in truth only 3 to 5 chances to 1 to be true”, the “truth” popping out of nowhere. (If you read French, the 300+ comments on the blog are also worth their weight in jellybeans…)
Share:
Like this:
8 Comments »