Rotating every certificate we own in 72 hours

It’s been six weeks since the OpenSSL heartbeat bug went public, and I finally have a quiet enough week to write up what we did. I’m not going to go into the bug itself again, I drew it out back in April. This is about the part afterwards, which nobody writes explainers for.

The patching went fine. The advisory came out on a Monday, and by Tuesday afternoon every box we knew about was on 1.0.1g, the processes linking OpenSSL had been restarted (we checked with lsof for anything still mapping the old library, and found two), and our load balancer provider had patched their side. If that had been the whole job I wouldn’t be writing this.

The problem is that patching only stops new leaks. Anything that sat in a vulnerable process’s memory over the previous two years might already be gone, and that includes the private keys for our certificates. Nobody was sure at first whether keys could actually be pulled out in practice. On Thursday, before anyone had proven it either way, we decided to assume the worst: replace every certificate we had, new keys for all of them, and revoke the old ones. By the weekend people had shown in public that keys really could be extracted, which made the decision feel a lot less paranoid.

The first thing I asked was how many certificates we had. Nobody knew.

Finding them

We started with the obvious list, the certificates someone could name off the top of their head: the main wildcard, the API endpoint, the marketing site. That was four. Then we went looking properly. We grepped every config management repo for ssl_certificate and SSLCertificateFile and .pem, we searched the shared password vault for anything that looked like a key, we went through about three years of receipts from our certificate authorities, and someone ran a scan across our public IP ranges and recorded every certificate that answered on 443 and a few other ports.

The receipts were the most useful source by far, which I wouldn’t have guessed. We ended up with 19 certificates. Some of them we’d forgotten about:

• a certificate on our VPN appliance, which uses OpenSSL and was vulnerable, and which nobody had thought of as “a website”
• one on a staging environment that a contractor had bought on a personal card in 2012 and expensed
• two internal certificates for service-to-service traffic, signed by our own little CA, whose key was sitting on one of the patched boxes
• one on an old status page from before we moved it off-site, still pointing at a box nobody had logged into in a year
• one that had expired in January, still installed, still served on a port nobody used

Nineteen isn’t many, I know. Bigger companies have thousands. But four known versus 19 real is the ratio that bothered me, and I’d bet it holds at bigger places too.

Reissuing

Reissuing was mostly tedious. Each one meant a new key and CSR, getting it through the CA’s site, waiting for validation emails, and deploying. Two of our CAs offered free reissues for this and handled it quickly. One took a day and a half to process ours, which was the main reason this took 72 hours and not 24. The status page certificate we just deleted along with the box.

The one that hurt was the API endpoint. Our mobile app pins it, and it pins the certificate rather than the public key. So a new key meant a new app release, and until users updated, the old app would refuse to talk to the new certificate. We ended up running the API on the new certificate behind a separate hostname, shipping an app update pointing at that, and keeping the old certificate, on a patched box, on the old hostname until enough people had upgraded. It isn’t pretty. It’s also the only part of this I’d call a design mistake rather than bookkeeping. If you pin, pin the public key and keep a backup key that’s never been deployed anywhere.

Our own internal CA was the other surprise. Its key had been sitting on a patched-but-previously-vulnerable box, so the whole CA had to go, and every internal certificate it had signed with it. That was a long Saturday.

Revoking, and whether it matters

Once a new certificate was live we revoked the old one, giving key compromise as the reason. I’m honestly not sure how much good it does. Adam Langley’s post last month about revocation checking makes the case that browsers’ soft-fail checks don’t protect you against anyone actually in a position to use a stolen key, and Chrome mostly doesn’t do online checks anyway. I’m persuaded by most of it. But revoking costs us nothing, and some clients somewhere do check, so we did it.

What really alarmed me came from Netcraft this month: tens of thousands of sites (around 30,000, by their count) revoked their certificates and got replacements issued with the same private key. That’s doing all of the work and none of the security. I can see exactly how it happens, too. Some CAs’ reissue flow defaults to reusing the CSR you already have on file, and if you’re tired and it’s the twentieth certificate, you click through. We caught ourselves about to do it once. After that, the checklist for each certificate had a line that said “compare the new public key fingerprint to the old one” and someone else had to tick it.

Afterwards

The inventory is now a file in the ops repo, one line per certificate with where it’s deployed, who bought it, which CA, the expiry date and the renewal process. There’s a weekly job that scans our public ranges and diffs the certificates it finds against that file, which is how we’ll catch the next contractor with a credit card. The internal CA got rebuilt with the key on a machine that isn’t on the network. And the mobile pinning is moving to public keys with a backup key, which needed its own app release and is going out this week.

We also found out how long it takes us to rotate everything, which I’d never known before. With the CA delay, it was about three days, plus another week of waiting for app upgrades. Next time I’d like the three days to be one.