My friends Isabella Verdinelli and Larry Wasserman published recently a new book with CUP in the spirit of the lean, terse, All of Statistics and All of Non-parametric Statistics (hilariously immodest titles!). This new volume is All of Regression, with expedited treatment of linear regression and hardly anything about machine learning (not even a mention in the index!). Time series barely escape oblivion within the final two pages!
(Our mutual friend) Ed George’s endorsement of All of Regression is worth quoting:
“Remarkable in scope and clarity, this book masterfully traces the evolution of regression from classical linear modeling to its many modern descendants. Each chapter offers a lucid introduction to a distinct methodology, explained with precision and insight from a contemporary perspective. Complete with exercises and datasets, it will serve equally well as a textbook and an indispensable research resource.”
The linear regression chapter shuns most of the linear algebra machinery swamping regression classics (except in its appendix), but it finds space to cover misspecified models, bootstrap, and model choice. The next and minuscule chapter is centred on X-validation for evaluating prediction error, while its follower is (barely) broaching on high-dimensional linear regression with a brief (and un-Bayesian) intuition on the Lasso. A few more words on logistic and Poisson regressions, and we are already landing at non-parametric regression.
The pace does not abate in any manner, though, since the univariate NP chapter covers kernels, local polynomials, k-nearest neighbours, and RKHS with a similar terseness (and no mention of Gaussian processes). Moving to multivariate regression zips thru trees and random forests, without severe consequences. Quantile regression is given the alm of a few pages (which imho may indeed be all it deserves!)
We are now already past the middle of the book, fighting for breath!, and classification then comes, mostly presented as an adaptation of earlier techniques to discrete outcome (which I find mildly objectionable for categorical data). With mentions of boosting (if not bagging) and SVMs. And no time for the ambiguity of the notion (which is at least as evil as mixtures, as earlier posted by Larry!) Of clear interest to the authors, the modern theory of conformal prediction gets its own, cosy, chapter, albeit somewhat misleadingly making it sound a very simple exercise. And no machine learning link. Next comes the equally modern and central notion of causality. With the (predictable) causal graphs, confounding variables, and instrumental variables (which have been a mystery to me since my first econometrics class). The final chapter is acknowledged as being made of parts each deserving an entire book, still getting a few pages at best. Empirical Bayes is restrained to linear regression, deep learning is questioned about being useful in statistical problems, and survival analysis mostly summarises as the Kaplan-Meier estimator.
Having gone through the book in a single rush (while on a train to Normandy), I felt more exhausted than at the finish line of a successful half-marathon (with a similar race time). It is great to see most fundamental notions linked with regression covered in such a minimalist way. With mathematical justifications whenever manageable in the alloted space. Albeit not expecting them (!), I regret Bayesian approaches to regression and classification not being mentioned. (No BART!) The computational aspects are not covered, hence readers may face a steep cliff when moving to actual problems but this is the theme for another course.
One funny typo: the bibliography (or reference list) is ordered by the first name, except for a misclassified Nicolai Meinshausen!, which presumes some first-hand knowledge of the entire community! And the inevitable “the the” makes its first (?) appearance on p12.
[The usual disclaimer applies, namely that this review is likely to appear later in CHANCE, in my book reviews column.]
