Archive for information

« Older Entries

mostly Monte Carlo [new season]

Posted in Books, Statistics, University life with tags , , , , , , , , , on September 9, 2026 by xi'an

The new season of mostly Monte Carlo has started with three talks this very Friday! At Paris Santé Campus as usual.

14h El Mahdi Khribch (ESSEC)

Contributions to the Theory of Bayesian Computation: Bias, Information, and Robustness.

Abstract: Bayesian inference is rarely computable exactly, and every practical substitute, whether a Monte Carlo sampler, a tempered posterior or a variational approximation, carries an error. This thesis gives finite-sample guarantees for three of them: the bias of sampling-based integration, the information cost of data-dependent posteriors, and the robustness of inference under misspecification. The unifying tools throughout are PAC-Bayesian change-of-measure inequalities and their information-theoretic counterparts.

15h Federica Milinanni (Northwestern University)

Rapid Mixing of Stereographic MCMC for Heavy-Tailed Sampling

Abstract: Sampling from high-dimensional, heavy-tailed distributions is a fundamental challenge in computational statistics, as many standard Markov chain Monte Carlo (MCMC) methods mix poorly in such settings. Recently, Stereographic MCMC [Yang et al., 2024] and the Sub-Cauchy Projection Sampler [Grazzi et al., 2026] have been shown to perform successfully on such tasks. However, establishing their non-asymptotic convergence properties remains an important open problem. In this work, we fill this gap by establishing non-asymptotic upper bounds on the mixing time of the stereographic projection and sub-Cauchy projection samplers. Our results demonstrate that, under certain conditions on the target and initial distributions, the mixing time is polynomial in dimension for a broad class of distributions, including light- and heavy-tailed cases.

Motivated by the theoretical analysis, we further establish a new weighted isoperimetric inequality that extends the classical version for (strongly) log-concave distributions to the heavy-tailed setting with optimal dimension dependence.

The proof techniques provide new insights into the geometric properties of heavy-tailed distributions that govern rapid mixing in high dimensions.

This is joint work with Tyler Farghly (Inria) and Jun Yang (University of Copenhagen)

16h Sylvain Procope-Mamert (INRAe)

A forward only method to construct proposal distributions in particle filters

Abstract: Particle filters are powerful algorithms used to sample from a sequence of distributions. It is useful notably, for Bayesian inference with different types of models and real data applications. In particular, when we try to recover a hidden signal from sequentially produced data with state-space models, the canonically defined proposals known as the bootstrap particle filter are rarely well-behaved and need extra work to be turned into useful sampling algorithms. Previous works on iterated methods for the automated construction of sequential Monte Carlo proposals, which were based on a backward scheme, have shown how to gradually improve proposals to reach a global optimality criterion, but they require a good initial proposal and cannot be used online.

Leave a comment »

the privacy fallacy [book review]

Posted in Statistics with tags , , , , , , , , , , , , , , , , , , , , , , , , , , , , , on May 3, 2024 by xi'an

“The World changed significantly since 1973.” (p.10)

I read this book, The Privacy Fallacy: Harm and Power in the Information Economy, by Ignacio Cofone, upon my return from Warwick the past week. This is a Cambridge University Press 2023 book I had picked from their publication list after reviewing a book proposal for them. A selection made with our ERC OCEAN goals in mind, but without paying enough attention to the book table of contents, since it proved to be a Law book!

“People’s inability to assess privacy risks impact people’s behavior toward privacy because it turns the risks into uncertainty, a kind of risk that is impossible to estimate.” (p.31)

Still, this ended up being a fairly interesting read (for me) about the shortcomings of the current legal privacy laws (in various countries), since they are based on an obsolete perception that predates AIs and social media. Its main theme is that privacy is a social value that must be protected, regardless of whether or not its breach has tangible consequences. The author then argues that notions that support these laws such as the rationality of individual choices, the confusion between privacy and secrecy, the binary dichotomy between public and private, &tc., all are erroneous, hence the “fallacy” he denounces. One immediate argument for his position is the extreme imbalance of information between individuals and corporations, the former being unable to assess the whole impact of clicking on “I agree” when visiting a webpage or installing a new app. The more because the data thus gathered is pipelined to third parties. (“One’s efforts cannot scale to the number of corporations collecting and using one’s personal data”, p.93) For similar reasons, Cofone further states that the current principles based on contracts are inappropriate. Also because data harm can be collective and because companies have a strong incentive to data exploitation, hence a moral hazard.

“Inferences, relational data, and de-identified data aren’t captured by consent provisions.” (p.9)

“AI inferences worsen information overload (…) As [they] continue to grow, so will the insufficiency of our processing ability to estimate our losses.” (p.75)

As illustrated by the surrounding quotes, the statistical and machine-learning aspects of the book are few and vague, in that the additional level of privacy loss due to post-data processing is considered as a further argument for said loss to be impossible to quantify and assess, without a proper evaluation of the channels through which this can happen and without a reglementary proposal towards its control. This level of discourse makes AIs appear as omniscient methods, unfortunately.

“Inferences are invisible (…) Risks posed by inferences are impossible to anticipate because the information inferred is disproportionate to the sum of the information disclosed.” (p.49)

“The idea of probabilistic privacy loss is crucial in a world where entities (..) mostly affect our privacy by making inferences” (p.121)

The attempts at regulation such as opt-in and informed consent are then denounced as illusions—obviously so imho, even without considering the nuisance of having to click on “Reject” for each newly visited website!—. De- and re-identified data does not require anyone’s consent. Data protection rights, as of today, do not provide protection in most cases, the burden of proof residing on the privacy victims rather than the perpetrators. The book unsurprisingly offers no technical suggestion towards ensuring corporations and data brokers comply with this respect of privacy and on the opposite agrees that institutional attempts such as GDPR remain well-intended wishful thinking w/o imposing a hard-wired way of controlling the data flows, with the “need of an enforcement authority with investigating and sanctioning powers” (p.106) . The only in-depth proposal therein is pushing for stronger accountability of these corporations via a new type of liability, with a prospect of class actions (if only in countries with this judiciary possibility).

[Disclaimer about potential self-plagiarism: this post or an edited version will eventually appear in my Books Review section in CHANCE.]

Leave a comment »

Bayesian restricted likelihood with insufficient statistic [slides]

Posted in Books, pictures, Statistics, University life with tags , , , , , , , , , , , , , , on February 9, 2022 by xi'an

A great Bayesian Analysis webinar this afternoon with well-balanced presentations by Steve MacEachern and John Lewis, and original discussions by Bertrand Clarke and Fabrizio Rugieri. Which attracted 122 participants. I particularly enjoyed Bertrand’s points that likelihoods were more general than models [made in 6 different wordings!] and that this paper was closer to the M-open perspective. I think I eventually got the reason why the approach could be seen as an ABC with ε=0, since the simulated y’s all get the right statistic, but this presentation does not bring a strong argument in favour of the restricted likelihood approach, when considering the methodological and computational effort. The discussion also made me wonder if tools like VAEs could be used towards approximating the distribution of T(y) conditional on the parameter θ. This is also an opportunity to thank my friend Michele Guindani for his hard work as Editor of Bayesian Analysis and in particular for keeping the discussion tradition thriving!

Leave a comment »

likelihood-free and summary-free?

Posted in Books, Mountains, pictures, Statistics, Travel with tags , , , , , , , , , , , , , on March 30, 2021 by xi'an

My friends and coauthors Chris Drovandi and David Frazier have recently arXived a paper entitled A comparison of likelihood-free methods with and without summary statistics. In which they indeed compare these two perspectives on approximate Bayesian methods like ABC and Bayesian synthetic likelihoods.

“A criticism of summary statistic based approaches is that their choice is often ad hoc and there will generally be an  inherent loss of information.”

In ABC methods, the recourse to a summary statistic is often advocated as a “necessary evil” against the greater evil of the curse of dimension, paradoxically providing a faster convergence of the ABC approximation (Fearnhead & Liu, 2018). The authors propose a somewhat generic selection of summary statistics based on [my undergrad mentors!] Gouriéroux’s and Monfort’s indirect inference, using a mixture of Gaussians as their auxiliary model. Summary-free solutions, as in our Wasserstein papers, rely on distances between distributions, hence are functional distances, that can be seen as dimension-free as well (or criticised as infinite dimensional). Chris and David consider energy distances (which sound very much like standard distances, except for averaging over all permutations), maximum mean discrepancy as in Gretton et al. (2012), Cramèr-von Mises distances, and Kullback-Leibler divergences estimated via one-nearest-neighbour formulas, for a univariate sample. I am not aware of any degree of theoretical exploration of these functional approaches towards the precise speed of convergence of the ABC approximation…

“We found that at least one of the full data approaches was competitive with or outperforms ABC with summary statistics across all examples.”

The main part of the paper, besides a survey of the existing solutions, is to compare the performances of these over a few chosen (univariate) examples, with the exact posterior as the golden standard. In the g & k model, the Pima Indian benchmark of ABC studies!, Cramèr does somewhat better. While it does much worse in an M/G/1 example (where Wasserstein does better, and similarly for a stereological extremes example of Bortot et al., 2007). An ordering inversed again for a toad movement model I had not seen before. While the usual provision applies, namely that this is a simulation study on unidimensional data and a small number of parameters, the design of the four comparison experiments is very careful, eliminating versions that are either too costly or too divergence, although this could be potentially criticised for being unrealistic (i.e., when the true posterior is unknown). The computing time is roughly the same across methods, which essentially remove the call to kernel based approximations of the likelihood. Another point of interest is that the distance methods are significantly impacted by transforms on the data, which should not be so for intrinsic distances! Demonstrating the distances are not intrinsic…

1 Comment »

Mea Culpa

Posted in Statistics with tags , , , , , , , , , , , on April 10, 2020 by xi'an

[A quote from Jaynes about improper priors that I had missed in his book, Probability Theory.]

For many years, the present writer was caught in this error just as badly as anybody else, because Bayesian calculations with improper priors continued to give just the reasonable and clearly correct results that common sense demanded. So warnings about improper priors went unheeded; just that psychological phenomenon. Finally, it was the marginalization paradox that forced recognition that we had only been lucky in our choice of problems. If we wish to consider an improper prior, the only correct way of doing it is to approach it as a well-defined limit of a sequence of proper priors. If the correct limiting procedure should yield an improper posterior pdf for some parameter α, then probability theory is telling us that the prior information and data are too meager to permit any inferences about α. Then the only remedy is to seek more data or more prior information; probability theory does not guarantee in advance that it will lead us to a useful answer to every conceivable question.Generally, the posterior pdf is better behaved than the prior because of the extra information in the likelihood function, and the correct limiting procedure yields a useful posterior pdf that is analytically simpler than any from a proper prior. The most universally useful results of Bayesian analysis obtained in the past are of this type, because they tended to be rather simple problems, in which the data were indeed so much more informative than the prior information that an improper prior gave a reasonable approximation – good enough for all practical purposes – to the strictly correct results (the two results agreed typically to six or more significant figures).

In the future, however, we cannot expect this to continue because the field is turning to more complex problems in which the prior information is essential and the solution is found by computer. In these cases it would be quite wrong to think of passing to an improper prior. That would lead usually to computer crashes; and, even if a crash is avoided, the conclusions would still be, almost always, quantitatively wrong. But, since likelihood functions are bounded, the analytical solution with proper priors is always guaranteed to converge properly to finite results; therefore it is always possible to write a computer program in such a way (avoid underflow, etc.) that it cannot crash when given proper priors. So, even if the criticisms of improper priors on grounds of marginalization were unjustified,it remains true that in the future we shall be concerned necessarily with proper priors.

3 Comments »

« Older Entries