Archive for heavy tails

escaping the dark side of the Moon

Posted in Books, pictures, Statistics, University life with tags , , , , , , , , , , , on March 18, 2026 by xi'an

Sub-Cauchy Sampling: Escaping the Dark Side of the Moon was recently posted on arXiv by Sebastiano Grazzi (Warwick), Sifan Liu, Gareth O. Roberts (Warwick), and Jun Yang. With an hommage to Pink Floyd’s 1973 album both Gareth and I listened to at the time. (This was for sure my first Pink Floyd album!)

This highly original work is a sequel to the stereographic projection paper by Yang, Latuszýnski and Roberts (which was itself vaguely connected to our unpublished origami sampler). As in the stereographic projection method, the Euclidean space supporting the target is turned into a spherical cap of a hyper-sphere, referred to as the complement of the dark side of the Moon (or its bright side), and defined with respect to an observer ο who was at the north pole in the original method. The proposed MCMC algorithm, the Sub-Cauchy Projection Sampler (SCS), is a random-walk-type Metropolis algorithm on the bright side and it gets its name from being uniformly ergodic for sub-Cauchy targets. An explanation for this massive achievement is that points at infinity in the Euclidean space are now mapped to the (d − 1)-dimensional boundary of the dark side rather than at the north pole of the hypersphere. Meaning that the push-forward density may remain bounded. (The random walk on the bright side involves projections for proposals ending on the dark side, while keeping the target intact.) There are several calibration parameters to the algorithm that can be tuned by variational arguments (and the goal of getting near a uniform distribution over the bright side), since optimal acceptance rates no longer apply.

All about that Bayes seminar [27 March, Paris Dauphine]

Posted in pictures, Statistics, Travel, University life with tags , , , , , , on March 25, 2024 by xi'an

The next All about that Bayes seminar will take place in Paris Dauphine, next Wednesday 27 March, at 2:00 p.m. (Room B Bis, 3rd floor) and will be delivered by François Caron (Oxford, visiting Paris Dauphine over the next two weeks) on

Deep Neural Networks with Dependent Weights: Gaussian Process Mixture Limit, Heavy Tails, Sparsity and Compressibility

This work studies the infinite-width limit of deep feedforward neural networks whose weights are dependent, and modelled via a mixture of Gaussian distributions. Each hidden node of the network is assigned a nonnegative random variable that controls the variance of the outgoing weights of that node. We make minimal assumptions on these per-node random variables: they are iid and their sum, in each layer, converges to some finite random variable in the infinite-width limit. Under this model, we show that each layer of the infinite-width neural network can be characterised by two simple quantities: a non-negative scalar parameter and a Lévy measure on the positive reals. If the scalar parameters are strictly positive and the Lévy measures are trivial at all hidden layers, then one recovers the classical Gaussian process (GP) limit, obtained with iid Gaussian weights. More interestingly, if the Lévy measure of at least one layer is non-trivial, we obtain a mixture of Gaussian processes (MoGP) in the large-width limit. The behaviour of the neural network in this regime is very different from the GP regime. One obtains correlated outputs, with non-Gaussian distributions, possibly with heavy tails. Additionally, we show that, in this regime, the weights are compressible, and some nodes have asymptotically non-negligible contributions, therefore representing important hidden features. Many sparsity-promoting neural network models can be recast as special cases of our approach, and we discuss their infinite-width limits; we also present an asymptotic analysis of the pruning error. We illustrate some of the benefits of the MoGP regime over the GP regime in terms of representation learning and compressibility on simulated, MNIST and Fashion MNIST datasets.