Finishing the data science certificate

I finished the data science certificate this year. It took a couple of years of evenings and weekends, mostly in R, and I’m glad I did it and also glad it’s over, in roughly equal measure.

Some context for why a guy who’s been writing software since he was 13 signed up for homework. Andrew Ng’s ML class in 2011 was the first time I’d seen machine learning explained in a way that clicked, and for years afterwards I was the person on the team who “knew some ML”, which mostly meant I could set up a logistic regression and explain why the accuracy was misleading. That’s a thin layer of knowledge over a big hole. I wanted the statistics underneath, done properly, with someone grading it.

What was worth it

The statistics, especially regression and all the assumptions hiding inside it. My math degree gave me the theory a long time ago and I’d lost most of it. Doing it again with real, messy data made it stick in a way that proofs never did for me. If I had to keep one thing it’d be regression, because it turns out that half the questions people at work bring me are regression problems once you squint.

The modeling habits, more than any particular model. I wrote about cross-validation in June, and the habit of not trusting a number until you know what data produced it is the thing I use most often. Random forests were the other big practical win, because they work well out of the box on the kind of tabular data a SaaS company has lots of. And doing backprop by hand again, which I complained about at the time, was worth the pain.

R itself, which surprised me. I came in thinking of it as a weird language statisticians put up with. I’m leaving it with the tidyverse and ggplot2 as my default for poking at any data set, and R Markdown as the way I write up anything with numbers in it. A C# programmer has a lot of complaints about R, and I have most of them, but nothing I know in .NET lets me go from CSV to a decent chart in four lines.

What I’d skip

Honestly, the parts that were about tools rather than ideas. There was a fair amount of time on setting up environments and specific packages, which matters on day one and goes out of date in a year. And I’d go lighter on the more exotic models. I spent a lot of hours on methods I’ve never used since, and probably won’t, because at our size the plain baseline usually wins and the hard part is getting clean data in the first place.

I’d also tell my past self to do fewer assignments at 11pm. The material didn’t get harder in the second year, but I got more tired, and I learned noticeably less from the courses I did while also running a team through a busy stretch at work. If you’re going to do one of these while working full time, budget the hours honestly. Most weeks it was two or three evenings and a chunk of Sunday, and in the weeks I tried to squeeze it into less than that, I mostly wasted the time.

What changed at work

We’re not suddenly an ML company, and I never expected that. But our A/B test reviews are better, because I now ask about sample sizes before the test starts instead of after. We replaced a hand-tuned churn score with a random forest that’s simpler to maintain and noticeably better in back-testing. And I’m a lot more useful in conversations with the data people we’ve been hiring, mostly because I can now tell when I don’t know something, which I couldn’t before.

It’s also changed what I read. A few years ago a stats paper would have been a skim, the abstract and the charts. Now I can usually follow the methods section, and once or twice I’ve even spotted the problem with one before the comments on Hacker News did.