Our ransomware tabletop

We picked the date back in March, mostly because it was the first afternoon in May when the CEO, our CFO and the two engineers who know our AWS setup best were all free at once. So it was luck, not foresight, that our ransomware tabletop ended up happening last Wednesday, five days after Colonial Pipeline shut down 5,500 miles of pipeline while gas stations across the Southeast were running out.

It did make it easier to get people to take it seriously. Nobody asked why we were spending an afternoon on this.

For anyone who hasn’t run one, a tabletop is a rehearsal done entirely by talking. Someone writes a scenario, and during the exercise they feed in new developments (“injects”) every fifteen or twenty minutes, and the people in the room say what they would do, who they’d call, and what they’d need. Nothing gets touched. You find out where the plan is imaginary. We’d done one before, a fake data breach for GDPR in 2018, and that one cost us 20 of our 72 hours arguing about who decides what counts as a breach. This one was worse, in a useful way.

The scenario

One of our senior engineers wrote it, and he made it nasty on purpose. On a Monday at 6:40 a.m., the on-call engineer gets paged because the app is down, logs in, and finds the production database instances deleted and a text file in several of our S3 buckets with a ransom note and an email address. Over the next hour, injects reveal that someone had phished an engineer’s Google login, including the second factor (through a proxied login page, which is the attack I said 2FA doesn’t stop, and which is why support got hardware keys last summer, though engineering still uses phone codes), and that since our AWS access goes through Google single sign-on, they’d been in our production account with admin rights for about four days. The note claims they’ve copied customer data and will publish it if we don’t pay.

Seven of us on a video call, three hours. Here’s what didn’t survive.

Backups were in the blast radius

This was the first thing we hit and the most serious. Our first instinct, all of us, was to restore. We have automated database snapshots and a nightly logical dump that gets copied into a separate bucket, and we’ve always talked about that bucket as “the offsite backup”. It’s in the same AWS account. The scenario’s attacker had admin in that account for four days, which means they could have deleted every snapshot and every object in that bucket, and an attacker who’s done this before would. If the attacker can delete a backup with the same credentials they used to get in, it’s really just a second copy.

We sat with that for a while. The fix is well known: copies in a separate account that the production credentials can’t touch at all, with S3 Object Lock on the bucket so even an admin in that account can’t delete them before they expire. We hadn’t done it because nobody had ever pictured an attacker with admin, only a fat-fingered engineer.

Break-glass was behind a door nobody had opened in a year

The next inject said the attacker had changed the passwords and MFA on every admin IAM user, which left the AWS root account as the only way in. The root account’s MFA is a hardware key, and the root password and backup codes are printed and in the safe, which I set up after the 2FA rollout and was pretty pleased with at the time. The safe is in our office. We went remote in March of last year, and hardly anyone has been in since. Two people know the combination, and one of them left the company in the fall.

I laughed, but it isn’t funny. In the scenario, getting into our own AWS account would have started with someone driving across town to an empty building and hoping the safe still remembered its combination. We’ve since moved the backup codes into a sealed envelope with two people who live in different places, and we’re registering a second hardware key for root. Neither is elegant.

We had nowhere to talk

The phished account was a Google login. Google is also our email, and Slack logs in through Google, and so does nearly everything else. Once the scenario said “assume the attacker has an admin session in Google Workspace”, the honest answer to “how are we coordinating” was: through the systems the attacker can read. Our CFO asked, reasonably, why we couldn’t just make a phone call, and that’s what we’d end up doing, but nobody had a list of everyone’s personal phone numbers that wasn’t in a Google Sheet. The annoying part is that I’d read a whole chapter on exactly this in Google’s security book last year, written about it here, and then done nothing.

During the exercise one of the engineers made a Signal group with everyone on the call, which is now the actual plan, and there’s a printed contact sheet in a few people’s houses.

Nobody knew who decides to pay

This was the part of the afternoon that got uncomfortable. The CEO assumed our cyber insurance would handle the ransom question. Our CFO pulled up the policy, and it turned out none of us had read it closely. It says we need to notify the insurer quickly and use an incident response firm from its approved panel, and we don’t have a relationship with any of them, so in a real incident we’d have spent part of the first morning picking a vendor from a PDF. It also doesn’t say the insurer pays, or decides.

Then there’s whether it’s even legal. Treasury’s OFAC put out an advisory last October warning that paying a ransom to a sanctioned group can itself be a sanctions violation, and that facilitators can be on the hook too. We’d want lawyers before any decision, and we don’t have a breach lawyer on retainer either.

Colonial’s own story came up a lot. Bloomberg reported on Thursday, the day after our exercise, that Colonial paid close to $5 million within hours of the attack, and that the decryption tool the attackers handed over was so slow that the company kept restoring from its own backups anyway. If that’s accurate, it’s the most useful thing I’ve read about ransomware this year. Paying didn’t get them out of doing the restore.

Nobody knew how long a rebuild takes

The last inject was the easy question: assuming the backups are intact, how long until customers can log in again? The database part we actually know. We’ve restored from backup once a month, into a scratch instance with a timer running, since the 2012 outage, and the drill says about three hours now. But the drill restores a snapshot into the same account, with the same credentials, and in this scenario the account is the attacker’s. Nobody had ever rebuilt the whole environment, app servers, secrets, DNS, the job cluster, into a clean account. The closest we’d come was the colo move six years ago. The app servers are built from configuration management, which helps, but the secrets they need live in the account the attacker owns, and someone pointed out we’d want to rotate all of them before trusting a rebuilt environment, and that list is long.

So the honest answer is that we don’t know. Somewhere between a long day and a long week. We’re scheduling a timed rebuild into an empty account next month to get a real number.

What I can’t settle

I’ve been chewing on the payment question since Wednesday, and I haven’t got anywhere. The fix for most of the above is engineering and paperwork, and it’s in the backlog with owners. If the backups had been in a separate locked account, I’d like to think we’d never pay, because a restore that takes days is survivable for a company our size. But the scenario had a second half, the threat to publish customer data, and a restore does nothing about that. We’d be paying to ask a criminal to delete something, and trusting them to do it. I don’t think I would. I’m not certain the CEO wouldn’t, and I’m not sure I’d be right to stop him. We agreed that the decision belongs to him, with legal advice, and that’s as far as we got.