The ex-employee who still had VPN access

A customer’s security questionnaire this spring had a question we’d been answering “yes” to for two years without really thinking about it: “Do you review user access to production systems at least quarterly?” We did review access, sort of, in the sense that I’d look at the list of people in our Google Apps account now and then. This time the customer wanted evidence, so one of our engineers and I sat down in June and did a real review, system by system, with a spreadsheet. It took a whole day, which was the first warning sign.

Most of it was fine. Then we got to the VPN.

What we found

Our VPN runs on the firewall appliance in our office, which also has a tunnel to our production network. When we set it up three years ago it was quicker to create local user accounts on the appliance than to hook it up to anything, so that’s what we did, and that’s how it stayed. Nobody’s offboarding process included logging into the firewall’s admin page, because nobody remembered the accounts were there. We’d been into that box’s admin page at least twice in 2014, to replace its certificate when we rotated everything after Heartbleed and to install the vendor’s fix once it finally shipped after Shellshock, and never once looked at its user list.

There were 26 accounts. We had 20 people who should have had one. Of the six extras, two were contractors from last year, two were people who’d left last year and never had production access anyway, one was a shared “test” account with a password I’d rather not print, and one belonged to an engineer who had left in November, seven months earlier, on good terms.

His account worked. We checked by asking him, which was an awkward phone call, and by looking at the appliance’s logs, which only went back 90 days. Within that window there was nothing. Before that we had no data at all. What we could see was that the account had the same access as any current engineer, which meant SSH to the bastion host, and from there to everything else if he still had his key in the bastion’s authorized_keys. He did. We’d removed him from Google Apps and from GitHub on his last day, and nothing else.

He was surprised and a bit sheepish on our behalf, and he said he hadn’t connected since he left, which I believe. The point is that we can’t prove it either way, and if his laptop had been stolen, or his credentials phished, we’d have had a stranger with a working path to our production databases and no way of noticing.

The rest of the afternoon

We disabled every account on the appliance that didn’t match a current person, rotated the shared test account out of existence, and went through the bastion’s authorized_keys line by line. That turned up four keys we couldn’t identify at all. Two turned out to belong to current engineers’ old laptops. The other two we never identified, so we removed them and waited to see if anyone complained. Nobody did.

Then we did the same thing for AWS, where we found an IAM user for the same ex-employee with an access key that AWS said hadn’t been used since it started tracking that in April, which is reassuring as far as it goes, and it doesn’t go back very far, plus two access keys for a monitoring service we stopped paying for last year. And then for the database, where several people who’d moved off the backend team still had write access to production. None of this was surprising once we started looking, which is sort of the problem. It’s the same lesson as when our host got breached and we found out how many keys we’d handed out: you don’t know what you have until you list it.

We wrote it up as an incident, even though nothing actually happened, and went through it the way we run our postmortems. That turned out to be the right call, because the conversation about “contributing factors” got to the real problem fast. There was no single place where a person’s access was listed. Every system had grown its own accounts, at different times, set up by different people.

The checklist

The VPN now authenticates against our directory, so disabling someone’s main account turns off the VPN with it. We’d like everything to work that way, and over the summer we’ve moved most systems over, but a few won’t, and those are on a list.

The list is the other half of it. We now have an offboarding checklist that names every system where an account or a key can exist, with an owner for each one. It goes something like this:

• directory and email account, and any mail forwarding
• VPN (now automatic, but checked)
• bastion authorized_keys and any other servers with personal keys
• AWS IAM users and access keys
• database users
• GitHub, our CI server and the error tracker
• third-party SaaS admin accounts: DNS, CDN, payments, support desk
• shared passwords they knew, which get rotated

The last item is the one people don’t like, because rotating a shared password means telling everyone else the new one. It’s also why we’re slowly getting rid of shared passwords.

The checklist is run by whoever manages the person leaving, on their last day, and the completed checklist gets filed with the HR paperwork, which sounds bureaucratic and takes about twenty minutes. And the full access review is now quarterly, for real, with the spreadsheet. We did a quick pass this month to check the fixes had stuck, which took about two hours, and the first proper repeat is in September. Our answer on that questionnaire is now true, which is a low bar, but we weren’t clearing it before.