Architecture & design · Book notes
Designing Data-Intensive Applications, read from the cloud side
The 2nd edition's new chapter 1 frames data systems around trade-offs, and its case for self-hosting with a predictable load sounds a lot like one machine at home.
How much have nine years of cloud changed a book that focuses on fundamentals? I was prompted to think about this because this spring we got the 2nd edition of Designing Data-Intensive Applications! Martin Kleppmann wrote the 1st edition in 2017, and since then it’s been the default recommendation for data-systems reading. For the 2nd edition he brought on a co-author, Chris Riccomini, and O’Reilly published it. They did a staged release, where you could read the early-release chapters for months before the print edition shipped.
The discussion around the release gives you a sense for how people relate to the book.. the top reply in the r/dataengineering release thread was a joke that most people will buy the new edition but still not read it, which was upvoted by over eight hundred people in the thread.
The new chapter 1 (“Trade-Offs in Data Systems Architecture”) introduces the book through the lens of three trade-offs: operational vs analytical, cloud vs self-hosting, and single node vs distributed. The section on cloud, which The Pragmatic Engineer ran as an excerpt, talks about how in cloud-native systems, local disks are “more like an ephemeral cache” and how you instead build on dedicated storage services like object stores. This in turn means that storage and compute get separated out into distinct pieces.
That same chapter doesn’t just say “the cloud is great” either.. it says that if you have an experienced team and a predictable load, it’s often cheaper to buy your own machines and run the software yourself, which stood out to me given the new cloud-heavy framing of the chapter. It makes it sound like a deliberate cost decision to self-host.
The chapter also talks about what operations looks like in the cloud era: instead of tending to individual machines, it’s about choosing services and migrating to new ones when your needs change. The rest of the book keeps the same spine as the first edition.. there are still chapters on replication and consensus.. but the chapter on indexing now includes vector indexes like HNSW.
Readers seem to disagree about how much it’s changed.. one person said Kleppmann called the edition fairly incremental and that first-edition readers don’t necessarily need to re-read it, but another person said right after the official release that it reads like an entirely new book. Some worry it’s become more cloud-centric and say they’ll keep the first edition around for a more old school view.
One long-running critique of the book is that while it’s a decent survey, it doesn’t provide the theoretical background you’d get from a university course on distributed systems. People defending the book say that the breadth is the point.. it’s useful for people who don’t know what they don’t know and need to make buy decisions. People who want more depth on storage engines tend to point to Database Internals.
The authors have also moved on from some things in the first edition.. in the BugBash podcast with both of them, they talk about how the new edition is framed around the move away from local disks and towards object storage and databases being rebuilt on top of S3. They also revisit the CAP theorem and suggest “offline availability” as a more useful way to think about that trade-off.
My homelab is the self-hosting case study for this chapter: predictable load, one machine at home. And the Stash archive already separates the record from the views: an append-only record in JSON sidecars, with notes re-rendered from it. I described that in Stash 2 the refresh engine that keeps my Reddit archive alive. So I’m left wondering: if the cloud treats local disk as cache, what would I gain (and give up) by doing the same with my own disk on a one-machine home setup?