The expired domain in our build

The watchTowr post went up on the 11th, and I read it on my phone that evening with growing delight, which is the wrong reaction to a security write-up but I’ll own it. The researchers noticed that the domain the .mobi registry used to use for its WHOIS server, before it moved to a new hostname, had been allowed to expire. They registered it for about $20, ran a WHOIS server on it, and within a couple of days were getting queries from well over a hundred thousand systems that still had the old hostname baked in: mail servers, security tools, government networks, and certificate authorities, at least one of which, they showed, could have been fooled into treating an address they controlled as the contact for a .mobi domain when issuing a TLS certificate. They stopped there, and the domain now points at a sinkhole.

What delighted me was how ordinary the mistake was. Nobody hacked anything. A hostname got written into a lot of software years ago, the organization behind it moved on, and the old name lapsed, because a domain registration is a bill, and bills for things you’ve stopped using get cancelled.

On Friday the 13th I asked the team a version of a question we’ve asked before. Which hostnames do our builds, deploys and dependencies reach out to, and do we know who owns each of them today?

Collecting the names

We already have most of the pieces. There’s the machine list from 2014, the agent list from after SolarWinds, the access-review sheet, which got billing columns after the Google Cloud weekend, and the vendor spreadsheet from Log4Shell. None of them is a list of domain names, though. A vendor’s name on a spreadsheet tells you nothing about the URL hardcoded in a script that someone wrote in 2017.

So an engineer and I went through the repos with a regex for anything that looked like a URL or hostname in Dockerfiles, CI configuration, shell and PowerShell scripts, Terraform, and the package manager configuration for npm, Bundler and NuGet. It was noisy, as expected. After removing our own domains, example.com, and the big obvious ones (the package registries, GitHub, AWS, Docker Hub, Microsoft), we had 63 registrable domains left. We looked each one up through RDAP, which is the modern replacement for WHOIS, and which felt appropriate.

Sixty-one were registered to who we expected and paid up for at least the next few months. One belonged to a vendor that had since been acquired, and the domain was renewed and redirecting to the new owner’s site, which is fine for now. And one had expired.

The one that had expired

It was in the Dockerfile for one of our older .NET services, the one that generates PDF reports. The PDF library we use needs a set of native font-rendering binaries, and years ago the vendor distributed those as a zip from their own download domain. Our Dockerfile fetched the zip with curl and unpacked it, with no checksum, from a URL on that domain. The vendor was bought in 2021 and the downloads moved to the acquirer’s site. The old domain was left to lapse this summer.

So why hadn’t the build been failing? Because of Docker’s layer cache. The step that fetched the zip hadn’t changed in three years, and our CI runners keep their cache, so every build since the domain stopped resolving had reused the layer it downloaded in 2021. If the cache had been cleared, the build would have failed loudly, someone would have fixed the URL, and that would have been the end of it. It’s an odd thing to say, but a failing build would have been the better outcome.

The RDAP record said the domain was in the redemption period, the window after expiry when only the previous owner can still renew it. After that comes a few days of pending deletion, and then anyone can register it. By our reading of the dates, that was about two weeks away. If someone had registered it then and put a zip at the same path, the next time a CI runner’s cache was cleared, or a new runner came up, or someone touched an earlier line in that Dockerfile, we’d have downloaded their binaries, with no checksum to stop it, into a service that has read access to customer report data.

We can’t register it ourselves until it drops, and I’m not sure we’d want to own a dead vendor’s domain forever. The fix was simpler. The zip now lives in our own artifact storage, at a version we’ve checked against the copy the acquirer still publishes, and the Dockerfile verifies a SHA-256 before unpacking. We found two other downloads in other Dockerfiles that also had no checksum, from domains that were fine, and gave them the same treatment.

The ones in the dependency tree

npm shows each package’s maintainers and their email addresses, so we pulled the maintainer list for every package in the main web app’s lockfile (it’s a bit over 1,400 packages now, up from 1,100 when I counted during event-stream), took the email domains, and checked those the same way. There’s a 2021 paper from NC State that did this for the whole registry and found a few thousand maintainers whose email domains had expired, so I knew we’d find something. The worry is the account recovery path: whoever controls the domain can receive the password reset email.

We found nine packages, all deep in the tree, maintained only by accounts whose email domains are expired or available. Some of those accounts may have two-factor authentication turned on, in which case a password reset alone shouldn’t be enough, and npm has required 2FA for its most-depended-on packages since 2022. But none of our nine are big enough to fall under that, and we have no way to see whether those accounts have it.

What makes this uncomfortable is the review comment we added in 2018. It lists new packages in a PR and who published them. If one of those nine accounts were taken over and published a new version, the comment would show the same maintainer name as always, which is exactly what a takeover looks like from the outside. For now, the nine are on a watch list, and any PR that bumps one of them gets a human look at the diff of the package itself, not just our lockfile.

On Friday we emailed npm’s support about the nine accounts, with the list. As of this morning we’ve had an automated reply, and nothing else.