Book review: Designing Data-Intensive Applications

Users would save a change, the page would reload, and the change would be gone. Then they’d hit refresh and it would be back. That bug took me most of a week, about ten years ago. You’ve probably guessed already: we’d just added a MySQL read replica, the save went to the primary, the reload read from the replica, and the replica was a second or two behind. I fixed it by routing reads to the primary for a few seconds after any write by that user, which I was quite pleased with. It was years before I found out that this problem has a name (read-your-writes consistency), a standard set of fixes, and a decade of papers behind it.

That’s what Martin Kleppmann’s Designing Data-Intensive Applications is good for. It finally came out in full this spring, after a long time in O’Reilly’s early-release program, and I’ve been reading it a chapter or so a week since May. It’s about 600 pages, and I’d happily have read 900.

The book is in three parts. The first is about a single machine: data models, how storage engines actually work, and how data gets encoded and how that encoding evolves over time. The second is about spreading data across machines: replication, partitioning, transactions, everything that goes wrong in distributed systems, and consistency and consensus. The third is about systems built out of other systems: batch processing, stream processing, and a final chapter where Kleppmann lays out where he thinks it’s all heading.

The storage chapter is where it hooked me. It opens with the world’s simplest database, two bash functions that append a key and value to a file and grep for the last match, and then fixes its problems one at a time until you’ve reinvented log-structured storage, then SSTables and LSM-trees, and then explains B-trees as the other branch of the family tree. I’ve used both kinds for years. I’d never seen them laid out so the trade-offs fall out of the design rather than being listed after the fact.

The transactions chapter is the one I’ve already made two people on my team read. It walks through what the isolation levels actually guarantee, as opposed to what their names suggest, and the example of write skew (two doctors both going off call at the same time, each one checking that the other is still on) is the clearest explanation of that anomaly I’ve seen. I went and checked which isolation level our main database runs at by default afterwards, and then spent an uncomfortable afternoon thinking about a couple of our “check, then update” code paths.

Chapter 8, The Trouble with Distributed Systems, is the one I’d hand to anybody who thinks distributed systems are just regular systems with more servers. Clocks drift, networks drop and delay packets with no way of telling which, and processes pause for seconds at a time in a garbage collector while the rest of the cluster decides they’re dead. His example of a lock holder that pauses, loses its lease without knowing, and then writes anyway, and the fencing tokens that fix it, is going to stay with me. Our job scheduler has exactly that bug, and there’s now a ticket to add fencing tokens to it.

Kleppmann is also refreshingly direct about marketing terms. He’s not a fan of the CAP theorem as a way of thinking about systems, and he points out that “consistent hashing” doesn’t have much to do with consistency, which made me feel better about my own post on it in January, because I’d never been sure what the “consistent” was supposed to mean either.

Complaints are few. The stream processing chapters in part three feel a bit more like Kleppmann’s personal views than the rest (he worked on Samza at LinkedIn, and it shows, in a good way mostly), and some of the material on the future of data systems is more speculative than I wanted after 500 pages of carefully argued trade-offs. The last section is on ethics, on what we owe the people whose data we collect, and I thought it was worth including in a book like this even if it reads differently from everything before it.

Every chapter ends with a huge list of references, a lot of them papers and blog posts I’d never have found on my own. That alone is worth the price. I’ve already got a stack of about fifteen printed out.

If you build anything that stores data, which is all of us, read it. If you’re a manager, read it anyway. It’s the fastest way I know to understand what your engineers are arguing about in design reviews, and to spot when they’re arguing about the wrong thing.