Social engineering at the help desk

How many people at your company could take over a customer’s account this afternoon, without the customer’s password?

I asked myself that on Wednesday evening, watching the Twitter accounts of Obama, Biden, Elon Musk, Bill Gates, Apple and a long list of others all post the same bitcoin scam within a couple of hours, send bitcoin to this address and I’ll send back double. It obviously wasn’t a few dozen people all reusing passwords on the same day. Twitter said that evening that it was a coordinated social engineering attack on its employees, aimed at people with access to internal systems and tools, and on Saturday it said 130 accounts had been targeted, the attackers had reset passwords and tweeted from 45 of them, and for up to eight they’d downloaded the account’s data. Motherboard had screenshots on Wednesday of what it said was Twitter’s internal account admin panel, and a claim from people involved that an insider had been paid, which Twitter hasn’t confirmed. How exactly the employees were manipulated hasn’t been made public as I write this.

So I don’t know the details, and nobody outside Twitter does. But the shape is clear enough. The most powerful login at a social network isn’t the president’s. It’s the support tool that can change the email address on the president’s account.

We have one of those. Every SaaS company does.

The admin tool

Ours is an internal web app, reachable over the VPN, that support uses all day. You search for a customer account, see their plan and billing history and users, and then there’s a row of buttons. Change the account owner’s email. Remove two-factor auth from a user who’s lost their phone. Reset a password. “Log in as this user”, which opens the customer’s view of the product so support can see exactly what they’re seeing. And export the account’s data, which exists because customers ask for it when they leave.

Every one of those is a legitimate thing for support to need. Losing your phone happens all the time, and the customer who can’t get in is upset and wants it fixed now. That’s exactly the pressure a social engineer uses.

On Thursday morning I sat down with the support lead and one engineer and we went through it the way the Building Secure and Reliable Systems book suggests: list every privileged action, then for each one ask who can do it, what they have to prove first, and what record it leaves.

Who can do it

The tool had exactly two roles, “admin” and nothing. Admin could press every button. 38 people had it. Support is eleven people, so the other 27 were engineers, two people in sales who’d asked for it to look up accounts during calls, a product manager, and me. Almost none of the 27 had used anything but the search box in months. They had full account takeover powers because the tool didn’t offer anything smaller.

We split it into three roles the same week: read-only (search and view, which is what sales and most engineers actually needed), support (the account-change buttons), and a small group of five who can export data and change the owner email on large accounts. The sales folks were fine with read-only. A couple of engineers grumbled, then admitted they’d never pressed the buttons.

What they have to prove

This was the part I was most nervous about, because it’s a process question, not a software one. When a customer calls or writes in and says “I’ve lost my phone, please turn off 2FA”, what does support check?

The written process said: the request has to come from the email address on the account, and the agent should verify two facts about the account, like the last invoice amount and the name of another user. Then they remove 2FA and the customer sets it up again.

The flaw is the first step. If someone has already taken over the customer’s email, which is the most common reason they’d be asking you to remove 2FA in the first place, “the request came from the right email address” proves nothing. And the “two facts” are sitting right there in the inbox the attacker has, since our invoices go to that address.

We pulled every 2FA removal from the past twelve months out of the tool’s database. There were 212. We read through 50 of the tickets behind them. Nearly all were obviously fine, a customer who’d changed phones and was on a screen-share with support while it happened. Three made us uncomfortable: the request came in from the right email, but from a different country than the account’s usual logins, the person was in a hurry, and the agent followed the process exactly. Two of the three accounts look normal since. The third customer churned a couple of months later, which probably means nothing, and we’ve emailed their old admin to ask.

The new rule for removing 2FA or changing the owner email is that the confirmation has to come through a different channel from the request. If the request is by email, support calls back the phone number we’ve had on file for at least 30 days, or asks another admin on the same customer account to approve it from inside the product. If there’s no other admin and no phone number, it waits 24 hours and the account’s owner gets a notice that a change is pending, with a link to cancel it. That delay is going to annoy some people who really have just lost their phone. I’ve decided that’s acceptable, and I’ve told support I’ll take the angry emails.

For the largest accounts, changing the owner email now needs a second person in that group of five to approve it in the tool, which is the multi-party authorization idea from the book, more or less unchanged.

What it records

Every button press already went into an audit table, which I was pleased about until we looked at it. It recorded who pressed the button and when. It didn’t record which ticket it was for, so there was no way to go from “this agent removed 2FA on this account in March” to why. Every privileged action now requires a ticket ID before the button works, and the tool checks that the ticket exists and belongs to that customer.

“Log in as” was the worst. It didn’t show up in the customer’s own audit log at all, so a customer had no way to see that someone from our side had been in their account. It does now, with the name of our staff member, and the session expires after 30 minutes.

The thing I can’t fix with a role

All of that is real progress, and most of it came out of one long Thursday and a couple of days of an engineer’s time. The gap that’s left is that every one of those controls assumes the attacker is outside, calling in as a customer. The Twitter story, as far as anyone knows, was the other way round: the attackers got at the employees, through their own credentials or by persuading them, and then used the tool exactly as designed.

Our support team has been working from home since March. They sign in to the admin tool through the VPN with their directory account and 2FA, and those 2FA codes come from an app on their phones, which a good enough phishing page can proxy in real time. Hardware security keys would fix that particular attack. We’ve ordered them for the eleven people in support and the five with export rights. But a key doesn’t help if someone calls a support agent at home, says they’re from our IT team, and asks them to “just approve this one change”, and I don’t have a technical answer for that. The best I have is that we told the whole support team exactly what happened at Twitter, as much as we know, and that nobody from our side will ever ask them to do that, and that if anyone does they should hang up and call me.