Book review: Building Secure and Reliable Systems

I’ve been reading this in the evenings for about a month, which says something about the length (550-odd pages) and something about my evenings right now. Google released it on its SRE site as a free download in early April, the same way it did the earlier SRE books, and O’Reilly sells the paper copy. The editors are Heather Adkins, Betsy Beyer, Paul Blankinship, Piotr Lewandowski, Ana Oprea and Adam Stubblefield, and there are dozens of other contributors, which explains both the best and the worst things about it.

The argument of the book is in the title. Security and reliability are usually separate teams with separate on-call rotations and separate vocabularies, and the book’s position is that they’re really one discipline, because they’re both about how a system behaves when something goes wrong, whether that something is a bad deploy or a person. A system that’s hard to understand is hard to keep up and hard to secure, for the same reasons. A system you can’t recover from a bad state is a reliability problem when the bad state is a corrupted config and a security problem when it’s an attacker’s backdoor. I mostly agree, and it’s a useful framing to have when you’re the one person at a small company who ends up owning both.

When I reviewed the first SRE book I complained that it was an anthology that kept repeating itself, with a long middle about Borg and Chubby you could skim. This one has the same problems, a bit worse. Some chapters read like they were written for a reader who already has Borg and a fleet of internal proxies, and the case study on running a publicly trusted certificate authority is interesting and of no practical use to almost anybody. But a few chapters are among the best things I’ve read on either subject, and they don’t need Google’s scale at all.

Least privilege

Chapter 5, “Design for Least Privilege”, is the one I’ve already used at work. The core idea is that most access should go through small, specific APIs that do one thing (restart this service, read this customer’s record for this ticket), instead of handing people a shell or a database login and trusting them to do only the one thing. Then the controls stack on top of that. Every privileged action needs a business justification, like a ticket ID, that’s checked rather than just logged. Some actions need a second person to approve them, which they call multi-party authorization, and the book’s case for it is refreshingly unparanoid: “Two pairs of eyes help avoid typos and mistakes”, and the insider-threat benefit comes almost for free. Access can be temporary, granted for an hour and then expired. And there’s always a breakglass path for emergencies, which is loud, heavily audited, and reviewed every time it’s used.

You don’t need Google’s scale for any of that. A 10-person company can put a “requires approval” step in front of its riskiest admin actions this week. We had a long discussion after I’d read this about which of our own admin actions would deserve two people, and the list was shorter than I expected and scarier than I’d like.

Recovery

Chapter 9, “Design for Recovery”, has the idea that surprised me most. Reliability engineering says that when a deploy goes bad you roll back, fast, to the last version that worked. Security engineering says some old versions are dangerous, because they have a known vulnerability, and an attacker who can trigger a rollback can bring that vulnerability back. So the book suggests tracking a minimum acceptable security version, a floor below which you refuse to roll back even if the newer version is broken. I’d never thought of rollback as an attack surface. We roll back all the time and it’s never once occurred to me to ask what we’re rolling back to, other than “the last green build”.

The same chapter is good on what it calls the cost of a recovery that’s rare: if you only rotate a key during an incident, then rotating it is a risky, manual, rarely practiced operation exactly when you’re stressed. That matched our experience after our host got breached years ago almost word for word. The fix it proposes is the one we arrived at, do the scary thing on a schedule until it isn’t scary.

Crisis management

Chapters 17 and 18, on handling a security incident and recovering from it, are the most practical in the book and the closest to a playbook. The part I’d underline is the advice about when to not act immediately. In an outage you fix things as fast as you can. In an intrusion, if you start cleaning up one compromised machine the moment you find it, you tip off the attacker before you know the extent of what they have, and they move or dig in. The book argues for scoping first, quietly, then remediating everywhere at once. That runs against every instinct an ops person has, and I think it’s right.

There’s also a decent section on communicating during an incident when you have to assume your normal channels are compromised, which I’d never considered and now can’t stop thinking about. If someone has your email and your Slack, where do the incident responders talk?

What I’d skip

The chapters on code and testing are fine but don’t say much you won’t find elsewhere, and the denial-of-service chapter is very much about defending a global network. The culture chapters at the end are earnest and a bit long. And because so many people wrote it, the voice changes from chapter to chapter, and some ideas get explained two or three times in slightly different words.*

If you’re short on time, read chapters 1 and 2 for the framing, then 5, 9 and 17. That’s maybe 150 pages, and it’s free.

* The first SRE book had the same problem, so I assume it’s a side effect of the format rather than a lack of editing. With six editors credited, it’s not for lack of trying.