The framework patch we hadn't applied

$ sudo find / -name 'struts2-core-*.jar' 2>/dev/null
/opt/tomcat/webapps/reports/WEB-INF/lib/struts2-core-2.3.16.jar

That’s the output I got last Wednesday night, on a server I’d forgotten about, a few hours after Equifax said which bug the attackers had used.

The background, if you’ve missed it: on September 7th Equifax announced that attackers had got at data on about 143 million US consumers, names, Social Security numbers, birth dates, addresses, and some driver’s license and credit card numbers. On the 13th they confirmed that the way in was CVE-2017-5638, a remote code execution bug in Apache Struts 2. That bug was disclosed and patched in early March. People were exploiting it in the wild within a couple of days of the announcement, because it’s about as easy as these things get: you put an OGNL expression in the Content-Type header of an HTTP request, the Jakarta multipart parser evaluates it while building an error message, and you’re running commands on the server. Equifax says the attackers were in from mid-May to late July. So, roughly two months after the patch, on a public-facing site.

It’s easy to be smug about that. I was, for about an hour. We have a dependency policy that I wrote, and that I’ve bored a few people with: critical vulnerabilities in anything internet-facing get patched within 7 days, and within 30 days for anything internal. We run automated checks on every repo’s lockfiles, and a bot opens a ticket when something we depend on gets a CVE. When Struts was in the news in March, I remember thinking it didn’t apply to us, because we don’t write Java.

Then I thought about it a bit harder, which is how I ended up running find on every box we have.

The jar is from a reporting tool that a contractor built for us about four years ago, when we needed some internal dashboards in a hurry and he knew Struts. It runs on Tomcat on a small VM. It’s only reachable over the VPN, which is the one thing that went right here. Its source lives in an old Subversion repository that never got moved to git with everything else, so none of our scanning had ever seen it. It wasn’t in the dependency inventory, because that’s generated from the repos. It had last been deployed in 2015, when it came out of the colo cage. It was on the machine list as reports-01, owner “ops”, which, as we learned during the colo move, means nobody. Version 2.3.16 is squarely inside the range that CVE-2017-5638 affects. So for six months we had an unpatched, trivially exploitable Struts app on our network, one VPN login away from anyone who got hold of a laptop, and our 30-day SLA would never have fired because, as far as our tooling was concerned, it didn’t exist.

I’d like to say we confirmed nobody touched it, but I can’t quite. Tomcat’s access logs don’t record the Content-Type header, which is where the exploit lives, so all I could do was look at who’d connected to the VPN and hit that box since March. It was three people, all of them employees, all of them looking at reports. The process list and the filesystem didn’t show anything odd. That’s not proof. It’s what we have.

The box is gone now. Two of the reports people actually still use got rebuilt as queries in the BI tool we already pay for, and nobody’s asked for the rest.

The full sweep turned up two other things the inventory didn’t know about. One was a Jenkins server whose plugins hadn’t been updated in about a year, because updating Jenkins plugins is unpleasant and nobody owned it. The other was a Docker base image for our background workers that was last rebuilt fourteen months ago, so it had fourteen months of Debian security updates missing even though the code on top of it had been deployed every week. That one bothers me more than the Struts app, honestly, because it was in plain sight. It was in a repo. Our scanner checks the repo’s Gemfile.lock and has no idea what the FROM line means.

The uncomfortable part is that our policy, the SLA I was proud of, was a policy about dependencies we knew about. Every check we had was downstream of an inventory, and the inventory was a list of repos someone had thought to register. The failures were all in things that had fallen out of the list, or that were never in it: a contractor’s project, a server nobody owned, the operating system layer under our own code. I suspect that’s what happened at Equifax too, though I’m guessing. They’ve said their security team knew about the vulnerability in March. Knowing about a vulnerability and knowing every place it lives are two very different things.

I don’t have a good answer yet for how you build an inventory of things nobody remembers. For now it’s a monthly find across every machine for a list of jar, gem and package names we care about, plus rebuilding base images on a schedule instead of only when someone remembers. It’s crude and it’s going to miss things. I’d love to hear how other small teams handle it, because I’d rather not learn the next one from the news.