A contractor's laptop and our identity provider

The screenshots came through our security channel at about 7 on Tuesday morning last week, from someone who’d seen them on Twitter before breakfast. They were posted by Lapsus$, the group that’s been leaking data from big companies for the last few months, and they showed what looked like an internal Okta admin console, with a support user’s view of a customer’s tenant and the buttons for resetting passwords and MFA factors.

Okta’s first answer, from its CEO that morning, was that this was related to an attempt in January to compromise a third-party support engineer’s account, and that the subprocessor had investigated and contained it. Later the same day its chief security officer, David Bradbury, published more. A support engineer working for Sitel, one of Okta’s support contractors, had a laptop that the attackers had remote access to for five days in January, the 16th to the 21st. Support engineers in that role can reset a user’s password and MFA factors, though Okta says they can’t download customer databases or create users. Okta’s worst-case estimate was that up to 366 customers, about 2.5% of them, might have had data viewed or acted on. By Friday Okta was saying, in so many words, that it had made a mistake by not acting sooner on the report it got in January.

I’m not going to say which identity provider we use. It matters less than you’d think, because every one of them has a support organization, and some of it is outsourced, and some of the people in it can do things to your tenant. Whether we were on the list was the first thing we asked, and the least useful. The better question was the one the screenshots raised: if somebody sat down at the laptop of anyone, anywhere, who can reset our employees’ MFA, how far would they get?

We’d done half of this before

In the summer of 2020, after the Twitter accounts were taken over through Twitter’s own internal tools, we went through our product’s admin tool and asked who could take over a customer’s account. That review held up well. The tool has proper roles now, removing 2FA or changing an owner’s email needs confirmation through a second channel, every privileged action needs a ticket ID, and “log in as” shows up in the customer’s own audit log. We checked those controls again on Wednesday and they were all still there.

What we hadn’t done, I realized when I sat down with our IT lead, was the same review for our own staff’s identities. All our attention in 2020 had gone on customers.

Who can reset us

So we made the list, the same way we’ve done every inventory on this blog since the host breach in 2013, by hand, in a spreadsheet, with a column for “why does this person have this”. Everybody in our identity provider who holds a role that can reset passwords or MFA factors, or change which factors a user has.

Seven names. Two were me and our IT lead, which is what I’d expected. Two were engineers who’d been made admins during the SSO rollout and never moved back to ordinary accounts, which was the kind of finding I’d also expected. The last three were people I’ve never met.

They work for the managed IT provider we brought in a while after we went fully remote, the company that ships laptops to new hires, wipes them when people leave, and runs the help desk for “I got a new phone and I can’t log in”. That last part is why they have the role. It’s a perfectly normal arrangement, and a lot of companies our size have exactly the same one. But put it next to the Okta story and it reads a little differently. Three people at another company, on laptops we don’t manage and have never seen, on a network we know nothing about, can reset any of our employees’ MFA, including mine, including the people with production access.

What we looked at

We pulled the identity provider’s admin log for everything those seven accounts had done since the start of the year, with extra attention on the January window, since if Sitel could be hit, so could anyone’s contractor. There were 31 MFA resets in the period. We matched every one to a help-desk ticket and, for the ones from the IT provider, to a message from the employee asking for it. All of them matched. As with every review like this, that means we found nothing, not that there was nothing, and I’ve come to accept that sentence as the honest end of most investigations.

We also talked to the IT provider, which was a better conversation than I’d feared. They were already getting the same question from other clients that week. Their technicians’ laptops are company-managed and they use hardware keys for their own admin access, they said, and they’re willing to show us how, which I’ll believe when they do.

The changes are mostly unglamorous. The two engineers are back to normal accounts. An MFA reset now sends a notification to the employee and to our security channel, so a reset nobody asked for gets noticed in minutes, not on the next audit. Resets for anyone with production access need a second approval from inside the company, with the person on a video call. And the IT provider’s role can no longer touch the admin accounts at all; if one of us is locked out, it’s the other one who fixes it.

What none of these changes touch is the laptop. Every one of them is about what happens after somebody acts. Okta, a company whose entire business is identity, had a contractor’s machine under someone else’s control for five days, and it took two months, and a Telegram post, for its customers to hear about it. If that’s what the vendor knows about its contractors, I don’t know what I’m supposed to know about my vendor’s contractors’ laptops, or how I would find out. Our security questionnaire asks about it. Everybody answers yes.