Zenbleed: a CPU bug explained

The machine under my desk at home has a Ryzen 7 3700X in it, which I bought for games and for compiling things, roughly in that order. As of yesterday afternoon it’s on the list of affected processors for a bug that lets any program running on it read fragments of what other programs on the same core are doing, at about 30KB a second per core, without any of the cache-timing tricks the 2018 bugs needed.

The bug is CVE-2023-20593, and Tavis Ormandy of Google, who found it, has named it Zenbleed. His write-up is very good, and terse in the way his write-ups always are. This is my attempt at the longer version, for people who remember Meltdown and Spectre but have never had a reason to think about what goes on inside the vector unit. I’m staying at the level of how the hardware works and where the mistake is. If you want the proof of concept, it’s on his site; I’m not going to walk through it.

Registers aren’t where you think they are

When you write assembly, or read a disassembly, you see a small fixed set of registers. On x86-64 with AVX2 there are sixteen vector registers, ymm0 to ymm15, each 256 bits wide. The lower 128 bits of each one is also addressable by its older name, xmm0 to xmm15, from the SSE days.

The chip doesn’t actually have sixteen 256-bit boxes with those names on them. It has a much bigger pool of physical registers, called the register file, and a table that says which physical register currently holds the value of each architectural register. That table is the register allocation table, the RAT. Every time an instruction writes ymm3, the CPU grabs a free physical register, writes the result there, and updates the RAT so that ymm3 now points at the new one. The old physical register goes back on the free list once nothing in flight needs it.

This is register renaming, and it’s how a modern out-of-order core keeps lots of instructions running at once without them tripping over each other’s registers. Two instructions that both write ymm3 but are otherwise unrelated can run in parallel, because they’re really writing two different physical registers. It’s been standard since the Pentium Pro, and if you drew the CPU pipeline in the branch prediction post at a finer grain, it’d be one of the boxes.

The important detail for Zenbleed is that the register file is shared by everything running on that physical core. Your process, the next process the scheduler puts on the core, the other hyperthread, the kernel. The RAT is what keeps them apart. If the RAT ever points your ymm register at a physical register that something else last wrote, you see their data.

vzeroupper

There’s an old performance problem with mixing SSE code (which only knows about the low 128 bits) and AVX code (which uses all 256). If the upper halves of the registers might contain something, the CPU has to preserve them across SSE instructions, which costs extra work. So the guidance has been, for a decade, that after you finish a block of AVX code you execute vzeroupper, which sets the upper 128 bits of every ymm register to zero. Compilers insert it for you. So does hand-written library code.

And there’s a lot of that library code. glibc’s strlen, memcpy and strcmp all use the vector registers on CPUs that support AVX2, because comparing 32 bytes at a time is much faster than one at a time. So a program that does nothing more than copy and compare strings is constantly writing data through the vector registers, and constantly calling vzeroupper at the end of those routines. Every HTTP header your server parses goes through them.

Zeroing sixteen registers by writing zeros to them would be wasteful, so AMD doesn’t. According to Ormandy, Zen 2 implements vzeroupper by setting a flag in the RAT, the “z-bit”, which means “the upper half of this register is zero, don’t bother looking”, and releasing the physical registers that held the upper halves back to the free pool. Nothing gets written to the registers themselves. That’s a nice optimization, as long as nobody ever needs to undo it.

Undoing it

Speculative execution means the CPU guesses which way a branch goes and runs ahead. If it guessed wrong, it throws away the work and rolls the machine state back to the branch. For ordinary instructions that’s well understood. The RAT keeps checkpoints and restores them.

Now suppose a vzeroupper sits on the wrong side of a mispredicted branch. The CPU runs it speculatively, sets the z-bit, and frees the physical register that held the upper half of, say, ymm0. Before the branch resolves, something else gets that freed physical register and writes into it. That something can be the other program’s strlen. Then the misprediction is discovered and the CPU rolls back. The z-bit is cleared, so ymm0’s upper half is “not zero” again, and it points at the physical register it pointed at before.

Except that physical register isn’t yours anymore. It was handed out and written while you were speculating. Your ymm0 upper half now contains whatever was last written there, by whoever.

It’s a use-after-free, in silicon. If you’ve done any CTF heap challenges you know the shape exactly: free something, let someone else allocate it, keep the dangling pointer, read. The only difference is that the allocator is the rename hardware and the pointer is a RAT entry.

Ormandy says making this happen reliably needs a few things to line up: the XMM register merge optimization, a register rename and a mispredicted vzeroupper, all inside a precise window. That’s the hard part, and it’s in his write-up for anyone who wants it. The point for the rest of us is that it doesn’t need privileges or any special system call. It’s ordinary instructions, so in principle any code that gets to run on the core can do it, and that’s why the cloud providers were the first ones patching.

Why this isn’t a side channel

When Meltdown and Spectre came out in 2018 I spent a long time drawing out how speculative execution leaks through the cache. Those attacks never read the secret directly. The speculative code touches memory in a pattern that depends on the secret, and the attacker then measures which cache lines are fast to load. It’s an inference from timing, and it needs careful measurement to work at all. Last year’s Hertzbleed was even more indirect, inferring data from changes in clock frequency.

Zenbleed skips all of that. The data lands architecturally in your register, and you just read it. No timing, no statistics. That’s why I think it’s a nastier class of bug than the speculative-execution family it superficially resembles. It’s a correctness bug in the rollback logic that happens to cross a security boundary.

How he found it

This part I enjoyed the most. Ormandy was fuzzing the CPU itself: generating random sequences of instructions and running them. The trouble with fuzzing a CPU is knowing when it’s wrong, since there’s no crash, just a wrong number in a register. His approach, which he calls Oracle Serialization, was to run each random program twice, once as generated and once with serializing instructions inserted between the instructions, so nothing can speculate or reorder. The two runs should end up with identical registers. If they don’t, the speculative machinery did something it shouldn’t have. He also used the CPU’s performance counters as a rough measure of whether a test case was exercising interesting parts of the pipeline, which is the hardware version of the coverage feedback that makes AFL-style fuzzers work.

I’d never thought of that, and it seems obviously right in hindsight, which is the best kind of idea. I’d be surprised if this is the last bug that technique finds, on anyone’s chips.

What to do

He reported it to AMD on May 15. AMD has microcode out now for the EPYC “Rome” server chips. For the consumer parts (Ryzen 3000 and 4000, some of the 5000 and 7020 mobile chips, Threadripper 3000) AMD’s bulletin lists firmware dates running into the end of the year, and that firmware comes through motherboard and laptop vendors, which means some machines will never get it.

There’s also a software workaround. Setting a “chicken bit” in a model-specific register (DE_CFG bit 9) turns off the behavior. That costs some performance, and how much depends on the workload. Ormandy notes that disabling SMT isn’t enough, since the register file is shared with whatever runs on the core, not only with the other hyperthread. I’d expect Linux distributions to start setting the bit on affected chips in their kernels if the microcode isn’t present.

For work, I went through our cloud inventory this morning. We don’t own any AMD hardware outside a couple of developer desktops, but some of our instances are on Zen 2 under the hood, and for those the fix is the provider’s job, at the hypervisor and microcode level. I’ve opened a ticket with our account rep asking for a clear statement on which instance families are patched, which I’d rather have in writing than infer from a status page.

For the 3700X under my desk, I’m going to set the chicken bit and see if my frame rate notices. I doubt it will.