Book review: Software Engineering at Google

I bought this when it came out in early 2020 and got about a hundred pages in before the world changed and I had other things to read. I picked it back up this spring because we’re having more and more arguments at work that sound like “do we really need to do it properly, we’re not Google”, and I wanted to know what doing it like Google actually looks like before I kept using it as a punchline.

Software Engineering at Google, curated by Titus Winters, Tom Manshreck and Hyrum Wright, with chapters by a long list of Googlers, is in three parts: culture, processes and tools. It’s about 600 pages, and it reads like what it is, an anthology. Some chapters are excellent, a couple are internal documents with the names filed off, and the voice changes every forty pages. It’s not a book you read straight through, and I’d suggest you don’t try.

The idea that holds it together

The opening chapter makes the distinction the whole book hangs on: software engineering is programming integrated over time. Programming is getting code to work. Engineering is what you do so that code keeps working while the people, requirements, dependencies and hardware around it change, for as long as it lives. A script you’ll run once and a service you’ll run for ten years are different problems, and most bad arguments about process, they say, are people with different time horizons in mind talking past each other.

That one idea did more for me than anything else in the book, because it resolves the “we’re not Google” argument better than either side of it. The useful question is how long this code has to live and how many people will change it, and size matters mostly because it changes those two answers. Our billing code is ten years old and has been touched by more than forty engineers. It should be held to a different standard than an experiment we’ll delete next month, and everyone on the team already knew that, we just hadn’t said it in those words.

What transfers

Hyrum’s Law gets its own section, and it’s the most useful page in the book for anyone with a public API. Hyrum Wright’s phrasing: “With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.” We learned this the hard way last year, when we changed the default sort order on an endpoint that had never documented one, and three customers’ integrations broke because they’d been relying on getting results oldest first. Deliberately randomizing anything you don’t promise, the way Go does with map iteration order, is a trick we’ve now copied for one of our list endpoints, so the next person to depend on an accident finds out in testing.

The Beyoncé Rule is the one I’ve been quoting. The gist is that if you liked it, you should have put a test on it: when an infrastructure team makes a change and your thing breaks, and there was no test in CI that would have caught it, the breakage is on you and not on them. At a company our size that sounds like it’s about Google’s scale, but it’s really about who owns what. Our platform team had been getting blamed for every upgrade that broke someone’s service, and the rule, more or less as written, has made those conversations shorter.

The chapter on deprecation is short and good, and it made me count. We have three different ways to send an email from our backend, and the oldest of them was supposed to be removed in 2019. Their point is that deprecation is work someone has to be assigned, not something that happens when you mark a function obsolete, and that the cost of keeping old systems around is paid by everyone who has to understand them. The leadership chapters were better than I expected, too. “Always be deciding, always be leaving, always be scaling” is a more useful summary of a manager’s job than most management books manage in three hundred pages.

The testing chapters’ small/medium/large split, by what a test is allowed to touch (one process, one machine, anything) rather than by unit and integration, is something we’ve adopted for naming our test suites. It took a couple of afternoons and it’s made it much easier to talk about why CI takes as long as it does.

What doesn’t

The tools section is the part I’d skim. It’s a tour of Google’s monorepo, its build system, its code search, its large-scale change tooling, and its static analysis, and it’s interesting the way a tour of a factory is interesting. Almost none of it applies to a company with one product and two dozen repositories. The one-version rule (exactly one version of every dependency, for the whole company, always) is elegant, and it depends on a level of build and test tooling we will never have, and I don’t think we should try.

Readability, Google’s system of certifying engineers in each language before they can approve code in it, I’d find stifling on a team our size, though I can see how it holds together across tens of thousands of engineers. And a few chapters, including the one on measuring engineering productivity, assume you have a research team whose job is studying your engineers, which is a nice thing to have.

Overall

I’d give the first ten or so chapters and the deprecation and testing chapters to every senior engineer on my team, and the rest to whoever asks. It’s a book about decisions you make once and pay for over years, written by people who’ve paid for them at the largest scale anyone has. At thirty people, we’re making most of the same decisions, just with fewer zeros on the bill, and it was a relief to find out the “we’re not Google” argument has a better answer than “yes we are” or “no we’re not”.