Book review: An Introduction to Statistical Learning

My copy has three different colors of sticky tabs in it and a coffee ring on the first page of chapter 6, which is about the most honest review I can give a textbook.

An Introduction to Statistical Learning, by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani, came out in 2013, and the authors put the full PDF on their website for free, which I didn’t know when I bought the paper copy (I don’t regret it, it’s a book you want to write in). It’s the younger sibling of The Elements of Statistical Learning, which Hastie and Tibshirani wrote with Jerome Friedman. I tried ESL a few years ago, after Andrew Ng’s online class, and bounced off it around chapter 3. It’s a great reference if you’re comfortable reading a page of matrix calculus as if it were prose. I’m not, anymore. My math degree is a long way back.

ISLR is the same ideas with the dial turned down on proofs and up on intuition. It covers what you’d expect: the bias-variance tradeoff, linear regression, classification (logistic regression, LDA, k-nearest neighbors), resampling with cross-validation and the bootstrap, regularization, splines and GAMs, tree-based methods, support vector machines, and unsupervised learning with PCA and clustering. Ten chapters, and from chapter 2 on, each one ends with a lab in R and a set of exercises, roughly half conceptual and half applied.

I’ve been working through it alongside the data science certificate, and it’s the book that made a lot of the coursework click. A few specific things it does better than anything else I’ve read.

The bias-variance chapter has a set of plots that I think about constantly: training error falling steadily as a model gets more flexible, test error falling and then turning back up into a U. I’d heard “overfitting” explained a dozen times. That one picture, with the same three simulated datasets carried through the chapter, is what actually stuck.

Chapter 6 explains why the lasso sets coefficients to exactly zero and ridge regression doesn’t, with a drawing: the lasso’s constraint region is a diamond, ridge’s is a circle, and the error contours are much more likely to first touch a diamond at a corner, where some coefficients are zero. That’s the coffee-ringed chapter. I’d memorized “lasso does feature selection” for an exam without understanding why, and it took one figure.

And the trees chapter is where I first properly understood bagging and random forests, which I wrote up with my own diagrams in July. The decorrelation argument in that post is basically theirs, with my churn data in place of their examples.

The labs are the other half of the value. Typing the code yourself, on the book’s own datasets (they’re in an R package, ISLR), and seeing the numbers match the text, is a lot more convincing than reading about it.

Where it’s showing its age: there’s almost nothing about neural networks, which in 2018 is a noticeable gap. The R in the labs is base R, lm() and glm() and square brackets everywhere, with none of the tidyverse that most R tutorials use now, so if you learned R recently it’ll look a bit old-fashioned. That doesn’t bother me much (base R is what you’ll find in other people’s code), but it’s worth knowing. And gradient boosting gets a few pages in the trees chapter, which undersells how much of applied machine learning on tabular data it’s become.

If you write software and want to understand what the data scientists on your team are actually doing, this is the book I’d give you. You’ll need some comfort with algebra and a little linear algebra, and you need to actually do the labs. If you’re past it and want the real math, ESL is waiting, and I might even try it again.