Use one big server

Anyone who has ever racked a server should read Use One Big Server, which went around last week. I agreed with nearly all of it, partly because it’s an argument I’ve been making in architecture meetings for about ten years and mostly losing.

The essay starts from a picture of a two-socket AMD server of the kind the big clouds buy: 128 cores, 256 threads, up to a terabyte of RAM populated cheaply and 8 TB if you pay for the dense DIMMs, and 128 lanes of PCIe 4, enough for a stack of NVMe drives and a 50 to 100 gigabit network card. Then it points out that one of these, rented from OVHcloud, is about $1,300 a month, and that the closest thing on AWS, an m6a.metal with 96 physical cores and 768 GB, is about $6,000 a month on demand. Its main claim is that if you have fewer than about 10,000 queries a second and you aren’t streaming video, one of these machines plus a backup in another data center is enough, and that a lot of the distributed architecture people build is paying for coordination they never needed.

I think that’s right, and I think most people who’ve run their own hardware would agree.

From about 2009 to 2015 we ran everything in a colo cage, first the whole product and then, after the main app went to EC2, the pile of things nobody wanted to move. Leaving it was the right call for us. But it wasn’t because the servers were too small. Our main database lived on one box in there, with a hot standby next to it, for the couple of years before the main app moved to EC2, and the only time it ran short of capacity was because of a missing index. What the colo cost us was everything around the servers: driving to the cage at 2 a.m. to swap a disk, and ordering hardware six weeks before we needed it and guessing at the size. And we had exactly one ops engineer who knew which cable went where. That’s the real cloud premium, and for a small team it’s worth paying.

What the essay gets right, and what I think people miss, is that you can have the cloud’s convenience without the cloud’s architecture. You can rent a very big VM and treat it like a server. That’s more or less what our Postgres primary is today, one large instance and a couple of replicas, and it’s the cheapest, least eventful part of our whole bill for what it does. The parts that cost too much are the ones built “cloud-natively” by people (me included) who assumed we’d need to scale out: a queue consumer split into four services that all deploy together, an autoscaling group that has never scaled, and a NAT gateway or two that nobody looks at until the bill does something strange.

The essay’s math on serverless is worth checking for yourself. It works out a Lambda equivalent for that m6a.metal at about 5.5 times the cost of the instance, and about 25 times the cost of the rented OVH box, and says that if you can keep a rented server above roughly 5% utilization, it’s cheaper than paying per request. I redid the arithmetic on a napkin and got the same ballpark. The exact factor depends a lot on your assumptions about memory per request, but it’s a multiple, not a percentage.

Where I’d push back is availability, and the essay is honest about it. It recommends a primary and a backup in different data centers, or two and two if you’re paranoid, and different hardware models so they don’t fail together. That’s fine advice, but it skips how hard it is to be sure the backup works. We had that hot standby in the colo, and the first time we really needed it, the replication had been broken for nine days and the alert for it went to a mailing list nobody read. One big server is simple. One big server plus a backup that actually takes over is less simple. A managed database with automatic failover is mostly paying someone else to have tested it.

So I’d put it like this. Run fewer, bigger things than your instincts tell you. Start with a monolith on one big box with a well-tested standby, and split things off when you can name the reason. Buy the managed version of the parts that are hard to operate well (databases, mainly) and rent plain big machines for everything else. And look at your current fleet. If you have twenty instances at 8% CPU, you might have one server’s worth of work spread across twenty places to page you.

We have at least three of those, and they’re the next thing on my list.