Archive for neural network

Privacy-preserving Computing [book review]

Posted in Books, Statistics with tags , , , , , , , , , , , , , , on May 13, 2024 by xi'an

Privacy-preserving Computing for Big Data Analytics and AI, by Kai Chen and Qiang Yang, is a rather short 2024 CUP book translated from the 2022 Chinese version (by the authors).  It covers secret sharing, homomorphic encryption, oblivious transfer, garbled circuit, differential privacy, trusted execution environment, federated learning, privacy-preserving computing platforms, and case studies. The style is survey-like, meaning it often is too light for my liking, with too many lists of versions and extensions, and more importantly lacking in detail to rely (solely) on it for a course. At several times standing closer to a Wikipedia level introduction to a topic. For instance, the chapter on homomorphic encryption [Chap.5] does not connect with the (presumably narrow) picture I have of this method. And the chapter on differential privacy [Chap.6] does not get much further than Laplace and Gaussian randomization, as in eg the stochastic gradient perturbation of Abadi et al. (2016) the privacy requirement is hardly discussed. The chapter on federated leaning [Chap.8] is longer if not much more detailed, being based on a entire book on Federated learning whose Qiang Yang is the primary author. (With all figures in that chapter being reproduced from said book.)  The next chapter [Chap.9] describes to some extent several computing platforms that can be used for privacy purposes, such as FATE, CryptDB, MesaTEE, Conclave, and PrivPy, while the final one goes through case studies from different areas, but without enough depth to be truly formative for neophyte readers and students. Overall, too light for my liking.

[Disclaimer about potential self-plagiarism: this post or an edited version will eventually appear in my Books Review section in CHANCE.]

Bayesian model averaging with exact inference of likelihood- free scoring rule posteriors [23/01/2024, PariSanté campus]

Posted in pictures, Statistics, Travel, University life with tags , , , , , , , , , , , , , on January 16, 2024 by xi'an

A special “All about that Bayes” seminar in Paris (PariSanté campus, 23/01, 16:00-17:00) next week by my Warwick collegue and friend Rito:

Bayesian Model Averaging with exact inference of likelihood- free Scoring Rule Posteriors

Ritabrata Dutta, University of Warwick

A novel application of Bayesian Model Averaging to generative models parameterized with neural networks (GNN) characterized by intractable likelihoods is presented. We leverage a likelihood-free generalized Bayesian inference approach with Scoring Rules. To tackle the challenge of model selection in neural networks, we adopt a continuous shrinkage prior, specifically the horseshoe prior. We introduce an innovative blocked sampling scheme, offering compatibility with both the Boomerang Sampler (a type of piecewise deterministic Markov process sampler) for exact but slower inference and with Stochastic Gradient Langevin Dynamics (SGLD) for faster yet biased posterior inference. This approach serves as a versatile tool bridging the gap between intractable likelihoods and robust Bayesian model selection within the generative modelling framework.

Doug’s scared…

Posted in Books, Kids, pictures with tags , , , , , , , , , , , , , on August 1, 2023 by xi'an

Following a link from a NYT editorial, I came upon a transcribed interview of Douglas Hofstadter on his fright about the current nature of AI and in particular of the impact of (surprise, surprise!) ChatGPT. When the French translation of Gödel, Escher, Bach came out in 1985, it became an immediate success, I read it, enjoyed it and recommended to my inner circles. To the point of my superior in the Navy (during the year I was drafted in the French Navy) saw the cover (for I was also reading it in the Navy office!), browsed through it and asked me to… reproduce the sculpture (a 3D ambigram) for the logo of his own company (for he moonlighted several days a week at an hologram company he had created). Which proved rather straightforward (but I ignore if the result was ever exploited). While I am now much less reserved about the book, which I feel is quite pretentious and self-congratulary, without delivering a particularly deep message, I can related to my earlier self excitement when faced with a scientific book involving many themes of interest for me, pleasant and easy to read, if sometimes mired in heavily making a point. This is obviously a personal view and others, like the science vulgariser launched a celebration for the 40 years of the book. (Coincidence: I had a chat & a beer with a former high school teacher of his’ in Normandy a week before writing this post.)

“it almost feels that way, as if we, all we humans, unbeknownst to us, are soon going to be eclipsed, and rightly so, because we’re so imperfect and so fallible.”

“it feels as if the entire human race is going to be eclipsed and left in the dust soon”

The tone of the interview is hilariously super catastrophic, foreseeing the replacement of humans by their “successors” within a few years, nothing less. There is no deep argument in the discussion, only that AIs are now playing chess and Go better than humans, can provide apparently reasonable answers to many questions, at a speed surpassing human faculties, including poetry or coding. Which proves such a reducing modelling of what constitutes a human being and the meaning of consciousness. (The last line of the interview “YouTube transcript cleaned up by GPT-4 & checked against audio” is hilarious, whether it is intended to be so or not.)

AIxcuse me?!

Posted in Statistics with tags , , , , , , , on May 3, 2023 by xi'an

posterior collapse

Posted in Statistics with tags , , , , , , on February 24, 2022 by xi'an

The latest ABC One World webinar was a talk by Yixin Wang about the posterior collapse of auto-encoders, of which I was completely unaware. It is essentially an identifiability issue with auto-encoders, where the latent variable z at the source of the VAE does not impact the likelihood, assumed to be an exponential family with parameter depending on z and on θ, through possibly a neural network construct. The variational part comes from the parameter being estimated as θ⁰, via a variational approximation.

“….the problem of posterior collapse mainly arises from the model and the data, rather than from inference or optimization…”

The collapse means that the posterior for the latent satisfies p(z|θ⁰,x)=p(z), which is not a standard property since θ⁰=θ⁰(x). Which Yixin Wang, David Blei and John Cunningham show is equivalent to p(x|θ⁰,z)=p(x|θ⁰), i.e. z being unidentifiable. The above quote is then both correct and incorrect in that the choice of the inference approach, i.e. of the estimator θ⁰=θ⁰(x) has an impact on whether or not p(z|θ⁰,x)=p(z) holds. As acknowledged by the authors when describing “methods modify the optimization objectives or algorithms of VAE to avoid parameter values θ at which the latent variable is non-identifiable“. They later build a resolution for identifiable VAEs by imposing that the conditional p(x|θ,z) is injective in z for all values of θ. Resulting in a neural network with Brenier maps.

From a Bayesian perspective, I have difficulties to connect to the issue, the folk lore being that selecting a proper prior is a sufficient fix for avoiding non-identifiability, but more fundamentally I wonder at the relevance of inferring about the latent z’s and hence worrying about their identifiability or lack thereof.