Auditing every agent with admin rights

Fourteen. That’s how many different pieces of third-party software we found running with full administrative rights somewhere in our infrastructure or on our laptops, root on Linux or SYSTEM on Windows. Before last week, if you’d asked me, I’d have said six or seven.

The week

If you’ve been away: on the 8th FireEye said it had been breached by a state-backed attacker who stole its red team tools. On Sunday the 13th it published what it had found while investigating, which was that the way in was a backdoor in SolarWinds Orion, the network and infrastructure monitoring platform. Someone had got into SolarWinds’ build process and inserted the backdoor into a DLL that shipped, signed with SolarWinds’ own certificate, in Orion updates released between March and June. The same day Reuters reported that the Treasury and Commerce departments had been breached, and CISA issued an emergency directive telling federal agencies to disconnect or power down Orion immediately. SolarWinds said in an SEC filing on Monday that fewer than 18,000 customers may have installed the bad versions. On the 16th FireEye, Microsoft and GoDaddy took over the domain the backdoor phones home to and turned it into a kill switch, and Microsoft said on the 17th that it had found the malicious Orion binaries in its own environment. A lot of the reporting attributes it to Russia’s foreign intelligence service, and the Secretary of State said on Friday it was “pretty clearly” the Russians, though there’s no formal government attribution yet.

The technical details in FireEye’s write-up are worth reading, because they explain why it went unnoticed for months. After installation the backdoor waits, for up to about two weeks, before doing anything. Its network traffic is made to look like Orion’s own telemetry protocol, the Orion Improvement Program. It stores what it learns inside Orion’s legitimate configuration files. And it checks for a long list of security tools and analysis environments and stays quiet if it finds them. This is people being patient and careful, against software that by design has broad credentials across a whole network, because that’s what a monitoring platform needs to do its job.

That last part is what got me. Monitoring software is the perfect target because we install it everywhere, give it more access than almost anything else, and let it update itself.

Did we have it?

No, and we could say so with some confidence by Monday morning, which was nice. One of our longer-serving engineers remembered that we’d trialed some SolarWinds products years ago, back when we were still in the colo, and every machine from that era has been gone since 2015. We searched our asset list, our software inventory and our cloud accounts anyway, and found nothing.

But “do we run Orion?” was the wrong question, and we all knew it by Monday lunchtime. The question was: what else do we run that has the same shape? Software from a vendor, running with full privileges, updating itself from the vendor’s servers, on lots of our machines at once. If that vendor’s build system gets compromised the way SolarWinds’ did, what does the attacker get?

The list

We did the same thing we did for 2FA in 2016 and for every machine we owned after Shellshock, which is open a spreadsheet and start typing. For each agent we wrote down what it is, who makes it, where it runs, what privileges it has, how it updates, what it can talk to on the network, and who on our side owns it.

On the servers: the infrastructure monitoring agent on every Linux host, the configuration management client, the log shipper, the cloud provider’s own management agent, the backup agent on the database servers, the vulnerability scanner’s agent, and the CI runners, which aren’t strictly an agent but run arbitrary code with deploy credentials, so they belong on the list. On the Windows build agents, the same monitoring agent again, running as SYSTEM. On laptops: the device management agent, the endpoint security product, the VPN client, the backup client, and the hardware vendor’s update utility on the Windows laptops, which runs as SYSTEM and which I’d never once thought of as software we run. That’s twelve we knew about, more or less. The other two were the reason for the exercise.

The first was an application performance monitoring agent, from a vendor we’d trialed in 2019 and didn’t buy, still installed and running as root on four of the production database hosts. The engineer who’d set up the trial left in the spring. The vendor’s trial account had expired months ago, so the agent was presumably sending its data to a dead endpoint, or maybe it wasn’t, and we couldn’t actually tell from our side. It came off all four hosts on Tuesday. It had been on our production databases for over a year with full privileges, and it wasn’t in anyone’s head.

The second was a remote support tool that one of our vendors had installed on a single Windows build machine to help debug a licensing problem two years ago. It ran as a service under SYSTEM, it was set to start automatically, and it accepted connections relayed through the vendor’s cloud. Nobody remembered agreeing to leave it there. I’ve seen this exact thing before, on a friend’s store computers in 2013, and I’d have told you we were past it.

Of the fourteen, ten update themselves directly from the vendor, with no staging step on our side, which means a malicious update would reach every machine they’re on within hours of the vendor shipping it. Eight can make outbound connections to anywhere on the internet, not just their vendor’s endpoints. And three were excluded from scanning by our own endpoint security, because the vendor’s install docs told us to exclude their folders and we did. SolarWinds has an article like that on its support site too, and so does nearly every vendor of this kind of software. It makes sense for performance. It also means the place a backdoor would sit is the one place we told our security product not to look.

What we’ve done so far

Some of it is easy and already done. The two orphans are gone. Each of the remaining twelve has an owner on our side, in the spreadsheet, who’s responsible for knowing what it does. The antivirus exclusions are gone wherever the vendor would support it, which is two of the three.

Egress is where we get the most for the least work. Most of these agents only need to talk to one or two vendor hostnames. On the servers we’ve started enforcing that with egress rules, so an agent that’s been tampered with can’t connect to some new domain it was never supposed to talk to. That wouldn’t have stopped SUNBURST on its own, since it was careful about blending in, but it closes the easier version of this attack, and it gives us logs to look at.

And wherever the agent supports it, we’re running it as a less privileged user instead of root. The log shipper only needs to read logs. Two of the agents support that already, and we’d simply never turned it on.

Staging updates, and why I doubt it helps

The obvious next step is to stop letting agents update themselves, pin versions, and roll updates out in stages: a couple of canary machines first, then everything a few days later. We’re doing that for the server agents.

But I keep turning it over, and I’m not sure it would have helped here. The bad Orion versions were signed, shipped through the normal channel, behaved normally for up to two weeks, and then blended in. A three-day canary would have shown exactly nothing. And for the endpoint security agent in particular, delaying updates means delaying the detection signatures and the fixes, which is the opposite of what you want from a security product. Microsoft pushed detections for the bad Orion DLL through Defender within days, and that only helps if updates aren’t stuck in our staging step. As the book I reviewed in May puts it in its own way, the things that make you reliable and the things that make you secure can pull in opposite directions, and this is one of them.

So for the agents I can’t stage, we’re trusting that each vendor’s build pipeline is better protected than SolarWinds’ was, and the only lever we have there is to ask. On Friday every one of those vendors got the same email from us: how are your release builds produced and signed, who can push an update to our machines, and would you tell us if your build system were compromised. Two have answered so far, one with a link to a security whitepaper from 2018.