Patching for Spectre and losing 30% on the database tier

The title says Spectre because that’s what everyone at work calls the whole mess, but strictly speaking this is about Meltdown, and specifically about KPTI, the kernel page table isolation patch. If you haven’t read it, I wrote up how Meltdown and Spectre work last month, and the short bit that matters here is that after KPTI, every trip from user space into the kernel costs more than it used to.

Our cloud provider patched its hosts in the first week of January, before most of us had finished reading the papers. That part we didn’t control and didn’t notice. The guest kernel was ours to patch, and our distro’s KPTI kernel landed the week after. I’d said in that post to take a baseline before touching the database boxes, so to be fair to myself, we did. We have a load test that replays a slice of a real weekday’s queries against a copy of the primary, and we ran it against the old kernel three times on the Friday before.

Then we patched one replica (which, it turned out later, was one of the worst-case boxes), ran the same test against it, and the number that came out was 31% fewer transactions per second at the point where latency fell apart. I ran it again because I didn’t believe it. 29%. In production, on the replicas we patched first, peak CPU went from about 52% to about 68%, and nearly all of the difference was system time, the time spent inside the kernel. The p99 on our busiest dashboard endpoint went from 38 ms to 61 ms.

The web tier, by comparison, barely moved. Maybe 3 or 4%. Which fits, once you think about what each one does all day.

Why the database

A database is about as syscall-heavy as ordinary software gets. For every query, Postgres reads the request off a socket and writes the result back to one. If the page it needs isn’t in its own shared buffers, it asks the kernel for it, and the kernel probably has it in the page cache, so that’s a cheap read that is now a less cheap read. On top of that there’s the background chatter, latches, signals, the WAL writer, the stats collector. We ran strace -c against a single busy backend for a minute and counted something like 30 syscalls for each query it served. Before KPTI, nobody cared. After, each of those means switching page tables on the way in and out.

The other thing that matters is PCID, a CPU feature that lets the kernel switch page tables without throwing away the whole TLB each time. With it, KPTI hurts. Without it, KPTI hurts a lot more. Our newer boxes had the pcid flag in /proc/cpuinfo. Two older ones, which had been sitting on an older instance type since we first built the tier, didn’t, and those were the ones at the 29–31% end. The newer ones were closer to 20%.

What we did about it

The cheapest fix was moving those two old boxes onto the newer instance type, which we should have done a year ago anyway. After that, the problem was really just “we now spend more per syscall, so make fewer of them”.

Our shared_buffers was set to 2 GB on boxes with 60-odd GB of RAM. Somebody set it that way years ago and it never got revisited, so Postgres was going to the kernel’s page cache for a lot of pages that could have lived in its own memory. We raised it to about a quarter of RAM, which is roughly what the Postgres docs suggest as a starting point, and the system time on the replicas dropped noticeably. That one’s on us. The patch only made the bill for it bigger.

The dashboard endpoint turned out to be doing an N+1: one query for the list and then one query per row for a count. For a big account that was 40-something round trips per page load, each one a few syscalls on our side and a few on the app server’s side. An engineer on my team rewrote it as two queries in an afternoon.

We’d also been letting each app process hold its own connections, which added up to several hundred Postgres backends at peak, most of them idle most of the time. Putting PgBouncer in front in transaction mode brought that down to a few dozen, which means less context switching and less memory, and it was overdue anyway.

And for two weeks, while all that went in, we just ran the primary on a bigger instance. It cost us a bit under $2,000 extra for the fortnight, which was cheaper than anyone’s time spent being clever about it.

With all of that, peak CPU on the primary is now at about 55%, a few points worse than December, and the p99 on the dashboard is lower than it was before the patch, mostly thanks to the N+1.

The argument we had

Somebody did suggest booting the database boxes with nopti, which turns KPTI off. The case for it is real: those machines only run Postgres and a couple of agents, no untrusted code, and Meltdown needs the attacker to be running code on the box. If someone can already run code on our database server, they probably have the database.

We didn’t do it. “Probably” was carrying a lot of weight in that sentence, and a code execution bug in an extension or a monitoring agent would go from “they can read what that process can read” to “they can read everything in memory, including the other tenants’ data in the buffer cache”. And a kernel flag that quietly disables a security fix is exactly the kind of thing that gets copied into the next machine image by someone who doesn’t know why it’s there. If we ever do it, it’ll be documented, on a box that’s isolated on purpose, with a reason written next to it.

The microcode side of all this, by the way, we never had to touch, because our provider handles it for VMs. Which turned out to be lucky. On January 22nd Intel told everyone to stop deploying its microcode updates because they were causing unexpected reboots on Haswell and Broadwell machines, and I know at least two people who spent that week rolling BIOS updates back.