Auditing our build images after the xz backdoor

Half a second. That’s roughly how much slower ssh logins got on Andres Freund’s Debian machines, and it’s the reason any of us know about this. He’s a PostgreSQL developer at Microsoft, he was doing some benchmarking, he noticed sshd using more CPU than it should and valgrind complaining, and instead of shrugging he dug in. On Friday he posted to the oss-security list that the upstream release tarballs of xz, versions 5.6.0 and 5.6.1, contained a backdoor. In his write-up, a failed login went from about 0.3 seconds to about 0.8.

I’m not going to go through how the backdoor works, partly because people far better at it than me have been taking it apart all weekend, and partly because you don’t need it to do the defensive work. The short description is that liblzma, the compression library from xz, isn’t something sshd uses directly, but on Debian- and Fedora-style distributions sshd is patched to talk to systemd, and libsystemd pulls in liblzma. The malicious code, which was only in the release tarballs and not visible in the git repository in the same form, hooked into sshd through that path at startup. By Saturday it was clear it was a way for whoever holds a particular private key to run commands on an affected machine, before any authentication.

The person who did this had been contributing to xz for about two years, under the name Jia Tan, and had become a co-maintainer, with what looks in hindsight like a pressure campaign on the original maintainer to hand over more of the work. I find that part harder to think about than the code, so I’ll leave it for now.

What “do we have it” means

Our first message in the channel, about an hour after Freund’s post, was the one we’ve sent every time since Shellshock: where is this, on everything we own?

The servers were the quick part. Our fleet is Ubuntu LTS and a few Debian stable boxes, and neither had shipped 5.6.x in a stable release. Debian’s advisory on Friday said the compromised packages had only been in testing, unstable and experimental. Our monthly package-name find, the one that started after the Struts jar in 2017, already knows how to ask the package manager for a version. I added xz-utils and liblzma5 to its list and ran it by hand on Friday afternoon. Everything came back 5.2 or 5.4.

The images were the slow part, and they’re why this took the weekend. Since Log4Shell our registry is the list we trust for “what actually runs”, so we pulled every tag pushed in the last 90 days, 412 of them across 28 repositories, and for each one asked whichever package database it had (dpkg, rpm or apk) what version of xz and liblzma it contained. That’s a short script and a long time waiting for image pulls. It finished on Saturday morning.

Where it was

Three places, none of them where I’d have guessed.

The first was a CI image. We have one that builds our docs and some of our generated client libraries, and about a year ago someone based it on debian:testing instead of stable because the docs toolchain needed a newer version of a couple of packages than stable had. Our base images are rebuilt every week on a schedule, a rule we’ve had since 2017 so that security updates don’t wait for someone to remember. That’s exactly why this image had a backdoored liblzma in it. It had been rebuilt on the 23rd, from testing, and picked up the new xz like it would pick up any other update.

It had no sshd. It runs for a few minutes per CI job and then goes away. Nothing anyone has published so far suggests the backdoor does anything without sshd loading it, and the code checks that it’s running inside sshd before it activates. So I don’t think it was exploitable as used. But I’m not going to stake much on “nothing published so far”, four days in. The image did, during each job, have a token that can publish to our docs site and to the package registries for the client libraries. We rotated those tokens on Saturday, rebuilt the image from stable with the packages it needed pinned from backports, and went back through the published client library versions since the 23rd to check they matched what the build logs said they should be. They did.

The second was laptops. Homebrew had shipped xz 5.6.x, and xz comes in as a dependency of a lot of other formulae, so it was on most of our engineers’ Macs whether or not they’d ever typed xz. Homebrew’s maintainers rolled it back to 5.4.6 on Friday, and the backdoor as analyzed only targets Linux on x86-64, built as a Debian or RPM package, so this one I’m fairly relaxed about. We asked everyone to run brew upgrade anyway, and checked with the same script against the laptop inventory on Monday. Fourteen Macs had it, and all fourteen are now on 5.4.6.

The third was the one that made me sit up. One of our engineers runs openSUSE Tumbleweed on his Linux laptop, which is a rolling release, and Tumbleweed had shipped the affected versions. He had sshd enabled, so he could get to the laptop from his desktop at home. It was only listening on his home network, there was no port forward on his router (we checked together, on a call, on Saturday morning), and the laptop has full-disk encryption. openSUSE’s advice was that anyone whose sshd had been reachable from the internet should reinstall, which his wasn’t. He reinstalled anyway, because it’s an afternoon and it’s an engineer’s laptop with access to things. He generated new SSH keys, re-enrolled the phone codes everyone in engineering uses, and we revoked everything his old keys could reach.

Scheduled rebuilds

So the thing I’m left with is that the policy that keeps us patched is the same policy that put the backdoor into our CI image within days. If we’d pinned that image and rebuilt it only when someone decided to, it would have been on an older, safe version of xz, and also on an older version of everything else, missing the security fixes we rebuild to get. The right answer is obviously “don’t base production-adjacent images on a testing distribution”, and that’s fixed. But the general version of the question doesn’t go away just because this instance did. Every weekly rebuild pulls in whatever upstream shipped that week, and we trust it because we trust upstream, and that trust is exactly what Jia Tan spent two years building.

One of the engineers who worked on this with me over the weekend asked, on Sunday night, how many other packages in our images are maintained by one tired person. We don’t know. He’s writing a script to find out, starting from the package lists we pulled for this, and I suspect the number will be large enough that it doesn’t tell us what to do.