cat posts/2014-09-30-the-shellshock-weekend.md

The Shellshock weekend

· 4 min read · security incidents bash devops

Last Wednesday, the 24th, the bash bug landed. CVE-2014-6271, quickly named Shellshock. If you set an environment variable whose value starts with a function definition, older bash keeps parsing whatever you put after the closing brace and runs it. Which would be a curiosity, except that a lot of things set environment variables from data an attacker controls. The famous one is CGI: a web server puts the incoming request’s headers into environment variables and then runs a script, and if that script is bash, or shells out to bash, the User-Agent header you sent is now code running on the box.

The test everyone was pasting that morning:

env x='() { :; }; echo vulnerable' bash -c 'echo hello'

If it prints vulnerable, you’re vulnerable. Ours printed vulnerable. So did basically everything.

I’ve written up a couple of these before, a host breach and Heartbleed. This one was messier, because the fix itself was moving under us.

Wednesday

We patched the obvious boxes Wednesday afternoon. apt-get update && apt-get install --only-upgrade bash on the Ubuntu fleet, the equivalent on the couple of CentOS boxes, re-run the test, get hello on its own, done. Our app isn’t CGI, it’s a long-running process behind a reverse proxy, so the most direct attack vector didn’t apply to our main stack. I went home thinking it was mostly handled.

Thursday

Thursday morning Tavis Ormandy posted that the fix was incomplete. There was a second way to trigger essentially the same parsing bug, filed as CVE-2014-7169, and the patch everyone had rushed out on Wednesday didn’t stop it. So the thousands of us who’d patched in a hurry got to do it again, and worse, got to explain to managers why “we patched it yesterday” was now “we patched the first one yesterday”. Over the next day or two more variants and more patches turned up, including an unofficial one from Florian Weimer at Red Hat that distros folded in. The upshot was that the version number you needed kept moving, and “are we patched?” stopped having a yes/no answer and became “patched against which of these?”.

The lesson I took from that is one I keep re-learning: when a fix is itself under active attack, stop treating “deployed the patch” as the end state and start re-testing on a schedule. We set the vulnerability test to run from our monitoring every hour, against every box, and alert if any of them went back to printing vulnerable. That caught two boxes the next day that had come up from an old image.

The weekend: the boxes we’d forgotten

The real work wasn’t the servers we knew about, it was finding the ones we didn’t. bash is everywhere. A short, incomplete list of what we ended up checking:

  • The web and worker fleet. Easy, config-managed, patched in minutes.
  • The CI workers, which run shell for every build and pull in untrusted-ish code from branches. These worried me more than the web servers.
  • Our monitoring box, which runs a lot of little check scripts in bash and, in a couple of cases, runs them against strings that come from external services.
  • Developer laptops. Most of us are on Macs, and Apple hadn’t shipped a fix yet, so Thursday and Friday everyone built bash from source following one of the dozen blog posts going around. (Apple did ship an official bash update yesterday, the 29th, for Mavericks and the two OS X versions before it. It took bash from 3.2.51 to 3.2.53 and wasn’t pushed through Software Update, you had to go download it, which I suspect means a lot of Macs out there are still vulnerable.)
  • The network gear. This is where it got uncomfortable. Our VPN appliance, a couple of managed switches, the out-of-band console server, all run some embedded Linux, and some of them have bash in there. For most of those there was no patch at all, just a vendor advisory saying they were “investigating”. All we could do was make sure none of their management interfaces were reachable from outside, which they shouldn’t have been anyway, and wait.
  • A Raspberry Pi under someone’s desk running an internal dashboard, which nobody could remember setting up and which turned out to be in our monitoring under a hostname like dashboard-temp. It had been “temporary” for over a year.
  • An old box running a cron job that one team depended on and that was on an OS release too old to get a bash update at all. We ended up building bash from source on it and scheduling it for decommission, which is the honest outcome: the bug didn’t create that risk, it just made us finally look at it.

We also went through our logs for the obvious exploit strings in the days around the 24th (() { :; }; in User-Agent and other headers) to see if anyone had hit us before we patched. We found plenty of scanning, which started within a day of disclosure, mostly mass scans probing /cgi-bin/ paths we don’t have. Nothing that looked like it landed. That’s the usual uncertainty, we found no evidence, which is not the same as there being none, but given we’re not running CGI I’m reasonably relaxed about it.

It came back to the inventory, again. After the host breach last year we wrote down every credential, after Heartbleed every certificate, and this weekend it was every box that can run a shell, which turns out to be a longer list than “servers”. The Pi and the cron box weren’t on any list because they weren’t “real” infrastructure, right up until a bug in the most common shell on Unix made them real. We’ve started a machine list now, with an owner column that I suspect is already wrong in places.

What still bugs me is the network gear. As of this morning the VPN appliance and the console server are running a bash nobody has patched, behind a firewall rule and a vendor advisory that still says “investigating”. I don’t know when that changes, and nothing on our side makes it go faster.