Rolling out 2FA everywhere in a week

Last Wednesday morning one of our engineers pasted a link into the ops channel with no comment, which is how you know it’s bad. It was Mathias Karlsson’s write-up, How I made LastPass give me all your passwords. The LastPass browser extension parsed URLs badly enough that a page on an attacker’s domain, with a carefully placed @ or two, could convince it that it was on twitter.com and get it to fill in your Twitter password. The day before, Tavis Ormandy had tweeted that he’d found a remote compromise in the Firefox add-on, and LastPass put out a fix for that one on Wednesday too. Karlsson’s bug had been reported and fixed a while ago, LastPass patched it in under a day, and as far as anybody knows neither was exploited.

We use LastPass, the whole company does, including a shared folder with the credentials for things that only have one login: the domain registrar, the DNS provider, the payment processor’s dashboard, the AWS root account. So we did the exercise you’re supposed to do after something like this. Assume someone has every password in that vault. What do they get?

The count

It took two of us most of Wednesday afternoon. We started from our quarterly access-review sheet, which had most of the services the company has an account with. It didn’t have a 2FA column, and it didn’t have the dozen services only sales knew about. With those added it came to 64 rows, and for each one we wrote down whether it supports two-factor auth, whether we have it turned on, and whether it’s enforced or just available.

It was worse than I’d guessed. Google Apps, which is our email and therefore the password reset for most of the other 63, had 2-step verification available, and 31 of our 52 people had turned it on. We’d never enforced it. AWS had MFA on the root account (I checked that years ago) but only on 9 of our 17 IAM users, and two of those without it had admin rights. GitHub was better, because most engineers turn it on for their personal accounts anyway, but there were still five people in the org without it, including one outside contractor. The registrar had no 2FA on our account at all, though it supports it. Neither did the DNS provider.

Put another way, someone with our vault and nothing else could have redirected our domain, read the CEO’s email, and spun up servers on our AWS bill. LastPass itself was protected by 2FA for most of us, which is good, but the whole point of the exercise was to assume the vault leaks some other way.

The week

On Thursday morning I sent an email saying that by the following Thursday, 2FA would be enforced on Google Apps, AWS, GitHub and everything in the shared vault, and that anyone who hadn’t set it up by then would be locked out until they did. Which is why I’m writing this today.

Most of the work wasn’t technical. Turning on enforcement in Google Apps is a checkbox with a grace period, and AWS is an IAM policy that denies everything except setting up MFA until you’ve done it. The shared accounts were the interesting part. A service with one login and one 2FA device has to live on someone’s phone, and then that person is a single point of failure, and they’re on vacation exactly when you need them. For the services that support several devices or backup codes we enrolled two people and printed the backup codes into the safe. For the ones that don’t, we stored the TOTP secret itself in a separate vault that only three of us can open. I’m not proud of that, since it puts both factors in the same kind of place, but it’s still a different vault with a different master password and it’s better than what we had.

The pushback was more than I expected, and some of it was fair.

Two people objected to putting a work app on their personal phone. I get it. We bought them YubiKeys, which work with Google and GitHub, so that was the end of that, at $18 each. One of our designers said it was security theater and that if LastPass was the problem we should stop using LastPass, which was a reasonable point and led to a good argument about what the alternative would be (it would be passwords in a spreadsheet, it’s always passwords in a spreadsheet). The sales team was mostly worried about getting locked out on the road. We printed backup codes for everyone and told them to keep them in their wallet.

And a couple of engineers argued against SMS as the second factor, since phone numbers can be hijacked by talking a carrier into moving them to a new SIM, which happens. NIST’s new draft authentication guidelines go further and call SMS “deprecated”. We allowed SMS anyway as the floor, because the gap between SMS and nothing is much bigger than the gap between SMS and an app, and pushed everyone toward the authenticator app.

By Tuesday we were at 49 of 52 on Google. The last three were one person on leave and two people who’d been ignoring email. Thursday morning we flipped enforcement on. Two people got locked out within the hour, one because she’d set it up on a phone she’d since wiped, and she was back in after ten minutes with an admin’s help. I don’t think anyone has noticed the AWS change except the two admins who’d been running without MFA.

What it bought us

I don’t want to oversell this. 2FA doesn’t help against malware on a laptop that’s already logged in, or against phishing pages that proxy the code through in real time, and the LastPass bugs themselves would still have leaked passwords. What it changes is that a leaked password, by itself, stops being enough. Given that LinkedIn’s 2012 breach turned out in May to be 117 million passwords, and Mark Zuckerberg’s Twitter account got taken over in June apparently from an old reused password, “by itself” covers most of how accounts actually get broken into.

The 2FA column is the part I’d recommend even if you never enforce anything. It’s in the access-review sheet now, and so are the sales team’s dozen services, and adding a new service means adding a row and filling it in.