Rotating the secrets in our password vault

A little over 300. That’s how many entries were in our company’s shared LastPass folders on Thursday afternoon, when LastPass published an update on its breach saying that whoever got in had copied a backup of customer vault data. As of this morning, a bit more than half of them have been rotated. This is a write-up of why we’re doing that, and why it’s slower than it sounds.

Three notices

We’ve used LastPass across the whole company for a long time. When two LastPass bugs landed in one week in 2016, we did the exercise of assuming someone had every password in the vault and asking what they’d get, and that’s how we ended up enforcing two-factor auth on everything. We didn’t leave LastPass then, and one of our designers told me at the time that we should. I’ll come back to that.

In August, LastPass said someone had got into its development environment and taken source code and some technical information, and that no customer data was involved. I read it, and we did nothing. On November 30 it said that information from August had been used to get into a third-party cloud storage service it uses, and that “certain elements” of customer information had been accessed. That one made me write a plan, but we didn’t act on it.

The December 22 update is the one that matters. By targeting another employee and obtaining credentials and keys, the attacker got into the cloud storage, decrypted some of the storage volumes, and copied a backup containing customer account information and a backup of customer vault data. LastPass doesn’t say how old the backup is.

What’s encrypted and what isn’t

The way a vault like this works is that your master password never leaves your device. The client runs it through a key derivation function, PBKDF2 here, many times over, to produce an encryption key, and uses that key to encrypt the sensitive fields with AES-256 before anything is uploaded. LastPass only ever stores the encrypted result. That design is sound, and it’s why LastPass can say the stolen vaults are still encrypted.

But “encrypted” only covers some of the fields. According to LastPass’s own notice, usernames, passwords, secure notes and form-fill data are encrypted. Website URLs are not. Neither is the account information next to them: names, email addresses, billing addresses, phone numbers, and the IP addresses customers connected from. So whoever has the backup can read, in plain text, a list of every site each of us has a login for. For our company’s vaults, that’s a fairly complete map of our vendors and our infrastructure, with the email addresses of the people who log in to each one. Even if they never decrypt anything, that’s a very good starting point for phishing us.

The encrypted part is a different kind of risk. The attacker has a copy, offline, forever. They can guess master passwords against it as fast as their hardware allows, without any rate limits or lockouts, and the only things slowing them down are the strength of each master password and the number of PBKDF2 iterations. LastPass says the default has been 100,100 iterations since 2018, and a 12-character minimum for master passwords. Wladimir Palant pointed out on Monday that existing accounts weren’t made to meet that minimum, and that some older accounts are still set to 5,000 iterations, which makes each guess about twenty times cheaper.

The thing I had to explain most often this week is that changing your master password now doesn’t help with any of this. The stolen copy is encrypted with whatever master password you had when the backup was taken. Changing it protects your account going forward, which is worth doing, but the attacker’s copy doesn’t change. The only way to make a stolen password useless is to change the password on the site it belongs to.

What we found when we looked

On Friday morning I asked everyone in the company to look up their own iteration count, which you can see in the account settings, and to tell me the number. Most were at 100,100. Eight were at 5,000, all accounts created before 2018. One of those was a vault I’d created myself in 2016.

That vault is the bad one. When we rolled out 2FA, some of our shared services only allowed one second-factor device, and for those we put the TOTP secret in a separate vault that only three of us can open. I wrote at the time that I wasn’t proud of it, since it puts both factors in the same kind of place, but that it was a different vault with a different master password. It’s also a LastPass vault, and its master password was fine, and its iteration count was 5,000. The other half of the problem is the secure notes in our shared folders. Over the years, copies of some backup codes had crept into secure notes in the same folders as the passwords they protect, the opposite of what we’d decided in 2016.

So for the most sensitive services we have, there’s a real chance the stolen backup contains both factors, protected by one master password at a weak iteration count. I don’t think anyone is likely to crack it soon. I’m not willing to bet the domain registrar on that.

The week

We’ve been going through the shared entries in order of how bad it would be if they were decrypted. First anything where the password alone is enough to get in, since 2FA covers most things now but not all of them. Then everything whose TOTP secret or backup codes were in a vault, which means turning 2FA off and enrolling again so the old secret stops working, plus generating new backup codes. That’s the slow part. Some of these services make you contact support to reset 2FA on a shared account, and support teams are, reasonably, not around this week. Then everything else.

A few of the entries had already been rotated in November, when we closed the access of the people we laid off. That was luck, not planning, and since LastPass hasn’t said when the backup was taken, we’re rotating those again anyway.

We also sent everyone a short note about phishing. Assume the attacker knows which services you use and your work email address, and expect very convincing emails about any of them. Our support team has used hardware security keys since 2020, which makes them hard to phish. Engineers use authenticator app codes, which a good enough proxy page can pass straight through. That’s been on my list for a while, and it’s moving up.

Personal vaults are each person’s own business, but we’ve told people what we’d do: rotate anything that matters, starting with email and banking, and if your iteration count was 5,000, treat the whole vault as exposed.

Where this leaves us

We’ll finish the rotation next week, when the vendors’ support teams are back. The question I haven’t answered is whether we stay on LastPass. The designer who said in 2016 that we should leave was partly right. The alternative we ended up arguing about that day, passwords in a spreadsheet, is still worse. The cryptography in most password managers is similar. What differs is whether the URLs get encrypted too, how the defaults are set and kept up to date for old accounts, and how quickly and plainly a company tells you something has gone wrong. On the last one, three notices over four months, the important one on the Thursday before Christmas, isn’t a record I’d pick.