Our host got breached, so we rotated everything

Last week the hosting provider for one of our smaller services told its customers that attackers had gotten into its systems, including customer data and credit card details. The cards were encrypted, but they couldn’t rule out that the keys had been taken too. Providers get breached and they were reasonably upfront about it, so I’m not going to pile on. What I want to write about is what happened when we sat down to answer one question: if an attacker has everything our provider had, what can they do to us?

The honest answer was that we didn’t know, because we’d never written down what our provider had.

Hours 0–2: the whiteboard

We grabbed a conference room and listed everything that touched that provider. It was a lot longer than any of us expected. There was the control panel login, which three people had, one of whom had left the team months ago. The company card on file. An API key our deploy scripts used to spin up boxes, with full account permissions because that was the only kind of key they offered. The root passwords on the servers, which, it turned out, we’d set at provisioning time through the provider’s panel, so the provider had them. Our SSH public keys, also uploaded through the panel, which is fine on its own except that anyone with panel access could add their own key the same way. DNS for two subdomains that we’d delegated to the provider’s nameservers years ago for convenience and then completely forgotten about. And database passwords and third-party API keys sitting in config files on the servers, which were only at risk if someone could get onto the boxes.

Which, through the provider’s out-of-band console, they possibly could. That’s the uncomfortable bit about a breach at your host. The console you use when you’ve locked yourself out is a back door to every machine you run, and it works for whoever controls the provider’s panel.

Hours 2–12: rotate, worst first

We ranked everything by how bad it would be in the wrong hands and worked down the list. Panel passwords first: everyone got a new one, the ex-employee got removed, and we turned on two-factor auth, which the provider had offered for ages and we’d never bothered with. Then the API key. We revoked it and issued a new one, and found out two scripts on a developer’s laptop were using it because they broke about ten minutes later. Then root passwords, which we reset directly on the servers this time, not through the panel, and a check of authorized_keys on every box. Nothing unexpected was there, which is a kind of relief you can only feel after you’ve actually looked. Then the card (the least scary item, and the one everyone asked about first), then moving those two forgotten subdomains back to our normal DNS provider.

Hours 12–48: assume the boxes are dirty

We couldn’t prove nobody had used the console to get onto our servers, so we went through the motions as if they had. We pulled the provider’s audit log of console sessions and our own auth logs and found nothing we couldn’t account for. We rotated the database passwords, our payment gateway key, the SMTP credentials and the S3 keys in the config files. That was the painful part, since every one of them meant a coordinated deploy, and two of them were embedded in places we had to go hunting for. The most sensitive box we just rebuilt from scratch, because that was faster than convincing ourselves it was clean.

We found no sign anyone had touched our systems. We’ll never know for sure, which is normal.

What we changed

We now have a single document listing every credential we own: where it lives, who can see it, what it unlocks and how to rotate it. It took a day to write, and next time the two-hour whiteboard session should take ten minutes. Root passwords and SSH keys get set by our own configuration management, never through a provider’s provisioning form. Offboarding now includes vendor accounts, not just the ones we happen to remember. The deploy scripts run under a separate sub-account that only has access to the servers they actually manage.

And once a quarter we pick one credential and rotate it on a boring Tuesday, just for practice. This time was the first time we’d ever rotated most of these, and you could tell.