Heartbleed, drawn out

A week ago the OpenSSL project put out an advisory for a bug in its implementation of the TLS “heartbeat” extension, and the internet spent the week in a mild panic. It got a logo and a name, Heartbleed, which tells you how bad it was considered to be. I’ve now explained it to non-programmers at dinner and to programmers at work, and the two explanations are different, so this is the programmer one. (The dinner one involves a librarian who reads you back as many pages as you ask for, whether or not your book has that many. It goes over surprisingly well.)

If you just want the to-do list: upgrade OpenSSL to the fixed version, restart every process that uses it (a lot of running services keep the old library in memory until you do), and then treat your server’s private keys and anything that passed through those services recently as possibly exposed. The password advice everyone gave was the easy, visible half. The private-key half is the one that’s going to keep security teams busy for a while, and I’ll come back to it.

What a heartbeat is

A couple of years ago TLS (whose public key half I drew out in the crypto post last fall) got an extension called the heartbeat. The idea is a keep-alive: one side sends a little “are you still there?” message with some payload in it, and the other side is supposed to echo the same payload straight back. It’s mostly useful for the UDP variant of TLS, where there’s no connection underneath to tell you the other end went away. For ordinary HTTPS it’s close to useless, which is part of why it stings that it was switched on by default.

The important detail is that the “are you still there?” message carries two things: the payload, and a number saying how long the payload is. The receiver is supposed to echo back that many bytes.

The bug, in one sentence

The receiver believed the length number without checking it against how much data had actually arrived.

If a message says “here are 65,535 bytes, echo them back” but only actually includes a few bytes of payload, a correct implementation notices the mismatch and throws the message away. The protocol spec even says so in plain language: if the claimed length is too big, silently discard it. OpenSSL didn’t compare the two. It took the length at its word, went to copy that many bytes back to the sender, and when it ran off the end of the small buffer that actually held the message, it just kept reading whatever happened to be sitting next to it in the process’s memory, and sent that back too.

Why it’s so bad

Most memory bugs I’ve chased, going back to Turbo C and pointers into the wrong segment, corrupt something and crash, which is loud. This one only reads and reports back, so nothing ever falls over. A server could leak for two years and look perfectly healthy the whole time, and this bug has been in every OpenSSL release since March 2012 (the code went in on New Year’s Eve 2011).

It also leaves nothing behind. A heartbeat is a normal, allowed message. There’s no failed login, no weird request in the access log, nothing that a normal setup would record. So after patching, almost nobody can say for certain whether they were hit, only that they could have been. That uncertainty is the expensive part.

The part that turned “annoying” into “change everything” is what sits next to the message buffer. It can be anything the process has been handling: other users’ requests, session cookies, and the server’s own TLS private key. When the bug first came out there was some hope that the private key specifically wouldn’t be reachable in practice. Within a couple of days people had demonstrated that it was, in a challenge set up for exactly that question. And a leaked private key is the bad case, because it isn’t something a user can rotate by changing a password. It means the certificate has to be reissued with a brand new key and the old one revoked, by every operator of every affected server, which is a slow, manual, worldwide chore.

The fix

The fix is exactly what you’d expect: before copying, check that the message is actually as long as it claims to be, and if it isn’t, drop it, which is what the spec said to do all along. It’s a couple of extra comparisons. That’s the uncomfortable thing about bugs like this. The patch is tiny and obviously correct once you see it, and the hole it closes was open for over two years in software that runs a large share of the secure web, maintained by very few people. The code had been reviewed. It passed everything. The missing check just wasn’t the kind of thing a test suite tends to look for, because you write tests for the messages your own code sends, and your own code never lies about the length.

What to actually do

For a normal person: change passwords on important accounts, but wait until the site in question has patched, because changing a password on a server that’s still leaking is pointless. Most of the big sites patched within a day or two.

For anyone running a server: upgrading the library isn’t the end of it. Restart the services so they stop using the old copy in memory. Then, because you can’t prove you weren’t hit, do the paranoid thing: reissue your certificates with new keys and revoke the old ones, rotate any secrets (session keys, API tokens, credentials) that lived in those processes’ memory, and consider forcing your users to log in again so old session cookies are worthless. That reissue-and-revoke step is a bigger job than it sounds once you count up how many certificates an organization actually has, which is a thing I’m in the middle of discovering at work right now. When I ran one of the public test scripts against our own boxes last Tuesday morning, every one came back vulnerable, including two I’d have told you didn’t run OpenSSL at all. That’s probably a separate post once the dust settles.