Archive for University of Oxford
maths as catchy background [and no further future for humanity]
Posted in Books, pictures, University life with tags AI, artificial intelligence, book reviews, dystopia, mathematical formula, neural networks and learning machines, Nick Bostrom, Philosophy of Science, poor maths, superintelligence, The Guardian, University of Oxford on May 11, 2024 by xi'an
All about that Bayes seminar [27 March, Paris Dauphine]
Posted in pictures, Statistics, Travel, University life with tags All about that Bayes, deep neural network, Gaussian processes, heavy tails, Lévy diffusion process, seminar, University of Oxford on March 25, 2024 by xi'an
The next All about that Bayes seminar will take place in Paris Dauphine, next Wednesday 27 March, at 2:00 p.m. (Room B Bis, 3rd floor) and will be delivered by François Caron (Oxford, visiting Paris Dauphine over the next two weeks) on
Deep Neural Networks with Dependent Weights: Gaussian Process Mixture Limit, Heavy Tails, Sparsity and Compressibility
This work studies the infinite-width limit of deep feedforward neural networks whose weights are dependent, and modelled via a mixture of Gaussian distributions. Each hidden node of the network is assigned a nonnegative random variable that controls the variance of the outgoing weights of that node. We make minimal assumptions on these per-node random variables: they are iid and their sum, in each layer, converges to some finite random variable in the infinite-width limit. Under this model, we show that each layer of the infinite-width neural network can be characterised by two simple quantities: a non-negative scalar parameter and a Lévy measure on the positive reals. If the scalar parameters are strictly positive and the Lévy measures are trivial at all hidden layers, then one recovers the classical Gaussian process (GP) limit, obtained with iid Gaussian weights. More interestingly, if the Lévy measure of at least one layer is non-trivial, we obtain a mixture of Gaussian processes (MoGP) in the large-width limit. The behaviour of the neural network in this regime is very different from the GP regime. One obtains correlated outputs, with non-Gaussian distributions, possibly with heavy tails. Additionally, we show that, in this regime, the weights are compressible, and some nodes have asymptotically non-negligible contributions, therefore representing important hidden features. Many sparsity-promoting neural network models can be recast as special cases of our approach, and we discuss their infinite-width limits; we also present an asymptotic analysis of the pruning error. We illustrate some of the benefits of the MoGP regime over the GP regime in terms of representation learning and compressibility on simulated, MNIST and Fashion MNIST datasets.
step-dads with Bayesian design [One World ABC’minar, 21 March]
Posted in Books, Statistics, University life with tags ABC, ABC World seminar, Approximate Bayesian computation, approximate Bayesian inference, Bayesian deep learning, Bayesian design, Bayesian learning, Canadian geese, experimental design, George Casella, One World ABC Seminar, statistical design, University of Oxford, University of Warwick on March 18, 2024 by xi'an
The next One World ABC seminar is taking place (on-line, requiring pre-registration) on Thursday 21 March, 9:00am UK time, with Desi Ivanova (University of Oxford), speaking about Step-DAD: Semi-Amortized Policy-Based Bayesian Experimental Design:
We develop a semi-amortized, policy-based, approach to Bayesian experimental design (BED) called Step-wise Deep Adaptive Design (Step-DAD). Like existing, fully amortized, policy-based BED approaches, Step-DAD trains a design policy upfront before the experiment. However, rather than keeping this policy fixed, Step-DAD periodically updates it as data is gathered, refining it to the particular experimental instance. This allows it to improve both the adaptability and the robustness of the design strategy compared with existing approaches.
(Which reminded me of George’s book on design in 2008.)
Masterclass in Bayesian Asymptotics, Université Paris Dauphine, 18-22 March 2024
Posted in Books, pictures, Statistics, Travel, University life with tags 2024, Bayesian asymptotics, Bayesian foundations, Bayesian inference, Bernstein-von Mises theorem, bois de Boulogne, course, empirical Bayes methods, foundation lectures, France, graduate course, IMS Lecture Notes, Judith Rousseau, marginal likelihood, MASH, Master program, masterclass, Paris, PariSanté campus, Université Paris Dauphine, University of Oxford on December 8, 2023 by xi'an
On the week of 18-22 March 2024, Judith Rousseau (Paris Dauphine & Oxford) will teach a Masterclass on Bayesian asymptotics. The masterclass takes place in Paris (on the PariSanté Campus) and consists of morning lectures and afternoon labs. Attendance is free with compulsory registration before 11 March (since the building is not accessible without prior registration).
The plan of the course is as follows
Part I: Parametric models
In this part, well- and mis-specified models will be considered.
– Asymptotic posterior distribution: asymptotic normality of the posterior, penalization induced by the prior and the Bernstein von – Mises theorem. Regular and nonregular models will be treated.
– marginal likelihood and consistency of Bayes factors/model selection approaches.
– Empirical Bayes methods: asymptotic posterior distribution for parametric empirical Bayes methods.
Part II: Nonparametric and semiparametric models
– Posterior consistency and posterior convergence rates: statistical loss functions using the theory initiated by L. Schwartz and developed by Ghosal and Van der Vaart, results on less standard or well behaved losses.
– semiparametric Bernstein von Mises theorems.
– nonparametric Bernstein von Mises theorems and Uncertainty quantification.
– Stepping away from pure Bayes approaches: generalized Bayes, one step posteriors and cut posteriors.
Asymptotics of ABC when summaries converge at heterogeneous rates
Posted in pictures, Statistics, University life with tags ABC, Approximate Bayesian computation, Bayesian consistency, COVID-19, curse of dimensionality, lockdown, PhD thesis, summary statistics, Université Paris Dauphine, University of Oxford on November 21, 2023 by xi'anWe just posted a new arXival, jointly with Caroline Lawless, Judith Rousseau, and Robin Ryder. This is a significant component of Caroline’s PhD thesis in Oxford, on which we started working during the first COVID lockdown. In this paper, we extend our results with David Frazier, Gael Martin, both with whom I’ll soon be reunited!, and Judith, published in Biometrika in 2018, to the more challenging case where different components of the summary statistic vector converge to their respective means at different rates, with some possibly not even converging at all. While this sounds impossible (!), we do prove consistency of the ABC posterior under such heterogeneous rates.
Wentao Li and Paul Fearnhead (also in Biometrika and in 2018) reduce the curse of the dimension of the set of summary statistic by showing, in the specific case of asymptotically normal summary statistics concentrating at the same rate, that a local linear post-processing step leads to a significant improvement in the theoretical behaviour of the ABC posterior. However, due to this focus on reducing the impact of the dimension of the summary statistics, it is therefore important to study its efficiency in a context where the summary statistics are not as well behaved. Surprinsingly maybe, we show that the significant improvement due to local linear post-processing persists even when summary statistics have heterogeneous behaviour. Most interestingly, the number of summary statistics which converge at the fast rate has no impact on the rate of posterior concentration nor on the shape of the ABC posterior (provided it exceeds the dimension of the parameter).
