Book review: Site Reliability Engineering

Most ops books I’ve read are either about tools, and out of date by the time they’re printed, or about culture, and vague enough to agree with without changing anything. Site Reliability Engineering: How Google Runs Production Systems (O’Reilly, April, edited by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy) is mostly neither, which is why I finished it, all 500-odd pages, over about three weeks of evenings.

It’s an anthology. Each chapter is written by different Googlers, and it shows: some chapters are sharp, some are padded, and the same ideas get re-introduced three or four times by people who clearly didn’t read each other’s drafts. You don’t have to read it in order, and I’d skip around.

The one-line definition is in Ben Treynor Sloss’s introduction: “SRE is what happens when you ask a software engineer to design an operations team.” The rest of the book is that idea worked out in detail. Google hires engineers into ops, caps their operational work, and expects them to automate their way out of the rest. The book says Google places “a 50% cap on the aggregate ‘ops’ work for all SREs”, and when a team goes over, the extra work gets pushed back to the developers who own the service. That’s a lot of institutional teeth, and it’s the thing a startup can’t copy directly.

What a startup can copy, and what I’d buy the book for if it had nothing else in it, is the error budget. The introduction puts it flatly: “100% is the wrong reliability target for basically everything.” Your users can’t tell 99.999% from 100%, because their own Wi-Fi and phone are much less reliable than that, and every extra nine costs you a lot. So you pick a target, say 99.9% for a month, which works out to about 43 minutes of downtime. Those 43 minutes are the budget. While you’re inside it, product teams can ship as fast as they like. When you’ve spent it, launches slow down until you’re back inside.

This sounds like bookkeeping, and the clever part is what it does to the argument. It’s the same tension The Phoenix Project, which I reviewed a few years ago, dramatizes as a war between dev and ops, except here it comes with a number. Every startup I’ve worked at has had the same fight between the people who want to ship and the people who get paged, and it’s always been settled by whoever argued louder that week. An error budget turns it into a number both sides agreed to in advance. I’d been trying to make this argument to our product folks for a year without a good way to say it, and now I just point at chapter 3.

The other chapters I’d single out:

The chapter on toil, which defines it as, among other things, work that’s manual, repetitive and automatable and that grows linearly with the service, is a good one to hand to a team that thinks it’s “too busy to automate”. The definition alone gave us a vocabulary for a conversation we kept failing to have.

The monitoring chapter’s argument is that pages should be about symptoms users can see, not causes, and that every page should require a human to do something intelligent. We had alerts on disk at 80% that paged someone at 3am to look at it and go back to sleep. We don’t anymore.

The postmortem chapter is about blameless postmortems, and their template is close to the one we’ve used since our cloud outage in 2012 (I wrote up how we run them last year). The one thing I’m stealing is a heading in their example postmortem, “where we got lucky”, next to what went well and what went wrong. Most of our near-misses have a line like that buried in the timeline, and it deserves its own section.

What doesn’t translate well: a lot of the middle of the book is about Google’s own infrastructure, Borg and Chubby and their load balancing, and unless you’re building a distributed lock service you can skim it. A few chapters also assume you have dozens of SREs per service and the organizational weight to say no to product teams. We have about six people who do any ops at all, and they’re also the people writing the features.

Also, the book is quite sure of itself. There’s very little about things Google tried that didn’t work, which is a shame, because I’d have learned more from those.

Still, I’d recommend it to anyone who carries a pager, and to their managers even more. Chapter 3 first, then toil, then monitoring, then whatever looks interesting.