Axios on npm was compromised, and our lockfile held

I read about it at 6:40 on Tuesday morning, on my phone, before I was properly awake, and the first thing I did was count the hours. The bad versions had gone up a little after 8 p.m. Eastern on Monday and come down a little after 11. Three hours, in the evening, on a Monday. Who on my team installs npm packages at 9 on a Monday night?

More people than you’d think, it turns out, but I’m getting ahead of myself.

What happened

Axios is the most popular HTTP client in JavaScript, with something over 100 million downloads a week. On Monday evening someone took over the npm account of one of its lead maintainers and used it to publish two new versions, 1.14.1 and 0.30.4. Neither contained any change to axios’s own code. What they did was add a new dependency, a package called plain-crypto-js, which had been published under a different account, first on Monday morning as a harmless copy of a real crypto library, and then again just before 8 that evening as a version with an install script.

So anyone whose npm install resolved to one of the new axios versions also installed that package, and npm ran its install script, as it does by default. According to the StepSecurity write-up, which is the most detailed one I’ve read, the script fetched a second-stage program for whichever OS it was running on, macOS, Windows or Linux, started it in the background so it outlived the install, and then tidied up after itself, deleting its own script and replacing the package’s manifest with a clean-looking one. A remote access trojan, in other words, on a developer’s laptop or a CI runner, with whatever credentials were on it.

npm pulled the axios versions about three hours after the first one went up. If you installed in that window, the advice is to treat the machine as compromised, and rebuild it and rotate everything it had access to.

The builds

Our main web app depends on axios directly, and about a dozen other packages in its tree depend on it too. That’s true of almost every JavaScript project of any size.

Our builds were never at risk, and I was able to say so by about 7:30. CI installs with npm ci, which installs exactly what the lockfile says, and has since event-stream in 2018. A quick search of every committed lockfile across all our repos found no trace of either bad version or of plain-crypto-js. Nobody’s lockfile had changed on Monday night.

The version-update PRs that Dependabot opens wait seven days after a release, the cooldown we added after the chalk and debug compromise in September. Axios 1.14.1 existed for three hours. It was never going to be proposed.

The agents were fine for the same reason. Our sandbox for unattended agent runs starts each container with npm’s before setting at seven days ago, so the registry looks to the agent the way it did a week earlier. That one mattered on Monday night, as it turned out. One of our engineers had started an agent at about 9:15, in the sandbox, on an internal tool that needed to call a new API, and the agent decided to add axios. It got the newest version from more than a week before, and the PR it opened on Tuesday morning had a perfectly ordinary axios in it. If he’d run the same task on his laptop, it would have pulled 1.14.1.

There was one more check that would have caught it. Since the fall, CI has flagged any package whose earlier versions were published with provenance, from a known build workflow, and whose new version wasn’t. Axios’s recent 1.x releases come from GitHub Actions through npm’s trusted publishing, and 1.14.1 had no provenance at all, because, as far as anyone can tell, it was published by hand with a stolen credential. That check never fired, because nothing got far enough to reach it, but I went and confirmed that it would have, which made me feel better about the afternoon we spent writing it.

The laptops

Which leaves the place none of that applies. Engineers install packages on their own machines all day, with credentials for our cloud account and GitHub sitting right there, and nothing between them and the registry but the lockfile, when there is one. Every engineer moved to hardware security keys in February, which stops a phishing page cold and does nothing at all about malware that’s already running on the laptop, with sessions that are already signed in.

Checking was harder than checking the builds. Because the malware cleans up its own manifest, looking for the version number in node_modules isn’t reliable. The write-ups agree that the sign it ran is the plain-crypto-js directory being there at all, whatever it claims to contain. So we pushed a small script through our device management to every company laptop, looking for that directory anywhere under a node_modules. It came back clean on all of them by noon.

But it was close in a way I didn’t like. Three people had run npm installs during the window, on Monday evening. Two of them were installing from lockfiles in our repos, which pinned an older axios and so were safe. The third was setting up a personal side project, on a personal machine, which isn’t ours to scan. He checked it himself, and it was clean, because his project didn’t use axios. He’d been planning to add it the next day.

None of that was a control, just three people’s evenings lining up the right way, so this week the seven-day cooldown goes onto the laptops too, through the npm config that’s part of every engineer’s setup, with the same exception for security fixes that we use in CI. Nobody likes it. Two people have already asked what happens when they need a release from yesterday, and the answer is that they ask, and someone looks at it first. That’s slower than what they have now by about ten minutes, and I’ve told them I’d rather argue about the ten minutes than about a rebuilt laptop.