Archive for NUTS
even faster HMC by learning leapfrog scale offline [online]
Posted in Books, Statistics, University life with tags Hamiltonian Monte Carlo, HMC, leapfrog integrator, No-U-Turn sampler, NUTS, population Monte Carlo, Statistics & Computing, tempering on September 24, 2026 by xi'an
faster HMC by learning
Posted in Books, Kids, Statistics, University life with tags algorithm, conformal Hamiltonian dynamics, eHMC, Hamiltonian Monte Carlo, HMC, leapfrog integrator, mirror descent, No-U-Turn sampler, NUTS, PhD thesis, population Monte Carlo, publication fees, revision, Scholarly Open Access, Shanghai, Springer Nature, Statistics & Computing, tempering, Università Ca' Foscari Venezia on September 23, 2026 by xi'an
Just received the good news that our paper Faster Hamiltonian Monte Carlo by Learning Leapfrog Scale by Wu Changye (吴昌烨), Pierre Pudlo, Julien Stoehr and myself, got accepted in Statistics & Computing! This is great in its own, but further concludes a story that started with Changye’s PhD thesis at Paris Dauphine in 2018, with a revision request from Statistics & Computing that stalled with Changye’s departing for industry in Shanghai and eventually resumed thanks to Julien’s massive investment in coding and improving the learning mechanism. It may also conclude my story with Statistics & Computing, where I am supposed to be the historically most prolific author (?), given the move by Springer to a cash-flow model on 01 January, 2027…
Nature tidbits [20 Aug 2026]
Posted in Books, Mountains, pictures, Statistics, Travel, University life with tags book review, borders, bullying, China academia, climate change, data analysis, Elon Musk, Erasmus, EU, Europe, exoskeleton, flood, Fuji-san, global warming, healthcare, human genome, Middle East, MMR vaccine, Nature, NUTS, ocean surface temperature, p-value, Pakistan, QED Science, sustainability, sustainable aviation fuel, The Economist, Trump 2.0, US Senate on September 16, 2026 by xi'anIn this 20 August edition, a wealth of data analysis papers, with many highly informative graphs. A call for China to better balance recognition for its junior scientists, another call for (US?) universities to treat bullying complaints in-depth rather than as a public relation exercise. The “compulsory” current status of the Trump 2.0 wreckage of US science with the Senate blocking the appointment of politicians for grant reviews. And with the absurd, ungrounded, health threatening split of the MMR vaccine. 
Another “compulsory” entry on climate change with a scary graph on the ocean surface temperatures rising way way above the confidence band represented by the previous measurements over 50 years. An interview (why?) of a CEO of the startup QED Science that claims an ability to automatically rank preprints and select the top 1% (why?!). Coïnciding with me receiving an email with a review by Claude of our Bayesian Adversarial Privacy paper! A report on the (defunct Starmer) UK Government cutting on fundamental physics projects. An article featuring the predicted rise in the use of exoskeletons, which I read on my way back from the lodge on Mount Fudji that was renting an exoskeleton to help with the climb! With an optimistic view on the consequences on people’s health and muscles. A short book review on Slobodian’s and Tarnoff’s Muskism, concluding that Musk(ism) lacks a coherent philosophy. (A Musk-read according to the Economist!) A long comment on the sleepless summer of the heatwaves, which led me clearly more tired than usual at the end of the summer break. (With a lot of should. And a focus on methodology.) Another one on the Middle East prospects as a sustainable aviation fuel (SAF) hub, which sounds paradoxical in the current geopolitic situation. It made me wonder how sustainable are these SAFs? True, they recycle oils, gases and biomass, rather than extracting “new” oils, but they still pollute the air. And, true, they can fill empty returning tankers, but they divert investment and research from more renewable and less polluting alternatives. Among the actual scientific articles, a long review on the safety and security of LLMs in healthcare (but useless pictures). 
One on the massive floods that occurred in Europe in 1342-1343, possibly caused by volcanic eruptions plus Arctic sea ice retreat, nothing that could occur today of course! With massive socio-economic consequences. (And a great graph, except I do not see the point for the 2D projections. Or for modern state borders.) 
An analysis of intergenerational mobility in Europe with the unsurprising conclusion that it impacts innovation. (Go, Erasmus!) With a multivariate regression I did not investigate (using NUTS but not Andrew’s NUTS!). And graphs of intergenerational mobility per European country, with too many colours to be readable (and the regression line to make sense?). Another data analysis of the Pakistan Genome Resource, with distinctive traits of the many ethnicities of Pakistan. And the discovery of many clinically relevant genes (with a massive avalanche of p-values!).
comments from Bob
Posted in Books, pictures, Statistics, University life with tags arXiv, Fisher divergence, Hamiltonian Monte Carlo, HMC, JMLR, leapfrog integrator, local adaptation, MCMC, NUTS, Riemann manifold, U-turn on April 10, 2026 by xi'an
Bob replied to my short post with further items of information that I find worth sharing:
Thanks for the kind post, Christian. It’s amusing to be the subject of one of these posts given how many of them I’ve read about other people. I really appreciate your summaries. And thanks to everyone in the audience for all the great feedback during and after the talk. Here’s a link to my slides.
One of your students or postdocs mentioned an approach that does continuous adaptation on some kind of polynomial schedule that is provably correct, but I didn’t manage to write down the author/reference or the name of the person who recommended it. If you happen to know what that is, I’d be grateful for the reference.
I would also like to follow up on the Robert & Andrieu paper you mention, but I could not find the exact reference on your Google Scholar page. The closest match I can find is:
Controlled MCMC for optimal sampling. 2001. C Andrieu, CP Robert. INSEE.
Section 1.3 is titled “Criteria for local adaptation.” The section cites two things. The first is Haario et al.’s (1999) sliding window approach, for which HMC moves too fast to be useful locally. The second is the multiple try approach of Liu et al. (2000) and the delayed rejection approach Tierny and Mira (1999). We applied delayed rejection to HMC step size adaptation in a couple of papers before developing GIST (Modi, Barnett and Carpenter in Bayesian Analysis; Turok, Modi, and Carpenter in AISTATS); these mirror our second GIST paper and third GIST paper in doing the step size adaptation for a whole trajectory and at each leapfrog step. The GIST approach is easier to understand, easier to describe mathematically, easier to implement, and is more efficient.
The nice part about GIST compared to Riemannian HMC is that we do not need to do any volume adjustments (which must be autodiffed through), which are cubic, and we do not need an implicit integrator, which is incredibly fussy to tune. The tradeoff is the we require reversibility of the adaptation, which I think is going to be tricky with varying curvature. Of course, we can’t afford to compute Hessian matrices in high dimensions, but we could manage Hessian-vector products if we could figure out how to use just those and we could also manage low-rank plus diagonal approximations or sketches as described in the Nutpie paper.
We’ve arXived the Nutpie paper since the talk:
Preconditioning HMC by minimizing Fisher divergence. arXiv. 2026. Seyboldt, Carlsen, and Carpenter.
The WALNUTS paper has been accepted by JMLR, but currently only the arXiv version is available:
The within-orbit adaptive leapfrog no-U-turn sampler. 2026. Nawaf Bou-Rabee, Bob Carpenter, Tore Selland Kleppe, Sifan Liu. 2025 arXiv; 2026 to appear JMLR.
Working with Nawaf and Tore has made all the difference in the world on this—it’s not something I could have done by myself. Sifan’s the one who came up with the nice characterization of NUTS and Nawaf’s done a number of additional things like providing mixing time bounds for NUTS (with Milo Marsden, who’s sadly no longer with us—he’s gone into finance).
Furthermore, you can adjust the U-turn criterion from 180 degrees to whatever you want to control how much of a full orbit you get. Those tend to be even more wasteful of iterations, though—this is what the plot from the expected integration time of NUTS is supposed to show, but it was confusing in the talk.
The approach you took with Wu Chengye to randomize number of leapfrog steps made a deep impression on me. It’s also wasteful in leapfrog steps because any number of steps greater than or less than about 1/4 of an orbit is wasteful either in computation or because it leads to more diffusive sampling. You can see that it is roughly as gradient efficient as NUTS in a 1000-dimensional standard normal. Interestingly, it’s worse than NUTS for parameter estimates and better for squared parameter estimates, which is overall a win. Nawaf has also published on randomized HMC. I think we could turn down NUTS U-turn criterion below 180 degrees to get something similar with NUTS, but I haven’t tried it.
One important property of your randomized approach is that it is much much easier to code efficiently for GPUs than NUTS, because the conditionals in NUTS are hard to execute in SIMD fashion. There’s a very nice introduction to this problem by Sountsov, Carroll, and Hoffman, in their paper “Running Markov Chain Monte Carlo on Modern Hardware and Software,” which is out on arXiv and also going into the next edition of the Handbook of MCMC). The thing to read about how to code NUTS on GPU is Dance, Glaser, Orbanz, and Adams’s paper, “Efficiently Vectorized MCMC on Modern Accelerators,” which is on arXiv and ICML 2025.
You can also randomize step size to vary the integration time and avoid harmonics, e.g.,
Randomized Hamiltonian Monte Carlo. 2017. Bou-Rabee and Sanz-Serna. Annals of Applied Probability.
Bob’s talk at PariSanté
Posted in Books, Statistics, University life with tags #ERCSyG, convergence of Gibbs samplers, Hamiltonian Monte Carlo, HMC, Mostly MCMC seminar, NUTS, Ocean, Paris, PariSanté campus, potential function, PSL, Sinkhorn algorithm, slides, Université Paris Dauphine, University of Warwick, WALNUTS on March 25, 2026 by xi'anWe had a wonderful time (and an unusually large audience) at the mostly Monte Carlo seminar last week as Pierre del Moral and Bob Carpenter both presented on exciting recent developments of theirs! Pierre talked about Kantorovich contraction of Markov semigroups, which sounds rather daunting!, but actually covers fairly general and generic convergence results, using tools like potentials and Lyapunov contractions, reminding me of the early days of MCMC and the papers of Gareth Roberts (University of Warwick), Jeff Rosenthal, Richard Tweedie and others.
Bob then spoke about the latest version of NUTS, the within-orbit adaptive NUTS (WALNUTS) sampler, which adapts the step size at every leapfrog step in order to conserve the Hamiltonian and keep the path stable enough. The adaptation is facilitated by incorporating this step size as an extra parameter with an attached distribution, that the authors call Gibbs self tuning (GIST), for coupling tuning parameters and conditionally Gibbs-sampling them per iteration in Hamiltonian Monte Carlo. This has been done in the past, incl. in some of my papers (e.g., Andrieu & Robert, 2004), but I could not cite a particular reference during the seminar.
Further light reflections that came to mind during Bob’s talk:
- with NUTS, if cycling is feasible in a finite time, we could wait for a second passage at the starting point and then get back halfway (with the difficulty of detecting this second passage)
- changing the kinetic matrix at each leapfrog jump is actually Riemannian HMC (and with cubic cost!)
- the doubling mechanism in both the original NUTS and in biased progressive NUTS is simulation wasting
- but so is (surprise, surprise!) finding adaptive mass matrices for WALNUTS at reasonable costs
