Two of our most-used npm packages got owned

“Is this us?” was the whole message, from one of our engineers at 1:52 yesterday afternoon, with a link to a GitHub issue on debug. I opened our lockfiles before I’d finished reading the issue.

If you were offline yesterday: someone phished the npm account of Josh Junon, who maintains a lot of very small, very popular packages, including chalk, which colors terminal output, and debug, which prints debug logging. The email came from a lookalike domain, npmjs.help, registered a few days earlier, and said his account would be locked unless he updated his two-factor settings. He entered his username, password and a one-time code on the fake page, which was all it needed. From a little after nine in the morning Eastern, new versions of about eighteen of his packages went up on npm, chalk 5.6.1 and debug 4.4.2 among them, with an obfuscated payload added. Together those packages get something like two billion downloads a week. The write-ups I’ve read say the payload only did anything in a browser, where it watched for crypto wallet activity and swapped the destination address on transactions. People noticed fast, Junon was posting about it within a couple of hours, and most of the bad versions were gone from the registry by the afternoon.

Every JavaScript project of any size depends on debug and chalk somewhere. When I counted last year, our main web app’s lockfile had a bit over 1,400 packages, and both are in there, several times, at different versions.

Where it got in

The main app was fine, and so was everything else that builds from a lockfile, which is everything we deploy. CI has installed with npm ci since event-stream in 2018, which installs exactly what the lockfile says and nothing newer. Nobody’s lockfile had moved yesterday morning, so nothing we built or deployed could have picked up the new versions. That took about twenty minutes to establish, by grepping every committed lockfile in every repo for the bad version numbers.

Then we looked at open pull requests, and found one.

An engineer had kicked off an unattended agent run at about 10 that morning, in our sandbox, on the internal command-line tool we use for running data migrations. The task was to bring its dependencies up to date and fix whatever broke, which is exactly the kind of job we use the long unattended runs for. The agent ran npm update, which does what it says and moves everything to the newest version allowed by the ranges in package.json. chalk is a direct dependency of that tool. So at around 10:20, an hour after the bad version went up, it was in a lockfile on a branch, and by 11:30 the agent had fixed two small breakages from other upgrades, run the tests, and opened a PR.

Our sandbox did what it was built for. It has no credentials and can only reach the registry, so whatever ran during that install had nothing to steal. And the tool is a Node program that never goes near a browser, so by everything I’ve read, the payload would have sat there doing nothing even if we’d merged it. But that’s luck about which package and which payload. The sandbox protects the machine the agent runs on. It does nothing about what the agent produces, and what this agent produced was a correct, tested PR with a compromised package in it.

Why it didn’t merge

Partly timing. The reviewer hadn’t got to it before the news broke. But we also have the bot comment from 2018, which lists every package that changed in a lockfile and who published it, and the comment on this PR listed chalk 5.6.1, published by qix. Which is what it always says, because that’s the maintainer.

Last September, after the expired-domain work, I wrote that if a maintainer’s account were taken over, the comment would show the same name as always, and that this is exactly what a takeover looks like from the outside. It’s a bit grim to see it happen this precisely. The comment didn’t help, and it couldn’t have.

What we’ve changed

A cooldown. Most of these attacks are found within hours or days, by people with better tooling than ours, so the cheapest defense there is is not being first. Dependabot added a cooldown setting in July, and our version-update PRs now wait seven days after a release before they’re opened. Security updates are exempt, because when there’s a real advisory we want the fix straight away, and I’d rather the urgent path be a deliberate one.

For agents, npm has had a before setting for years, which makes the resolver act as if nothing published after a given date exists. The sandbox now sets it to seven days ago when the container starts, so an agent running npm update sees the registry as it was a week ago. That’s a three-line change to the container’s startup script. It means an agent can’t pick up a fix released this week, either, and I’m fine with that. That’s what the security-update path is for.

Provenance, which I’m less sure about. Since the end of July npm has had trusted publishing generally available, where a package is published from a CI workflow with a short-lived token instead of from someone’s laptop, and it comes with a signed record of which repository and workflow built it. npm audit signatures checks those, and it’s now in CI. What I really want is a check that flags a package whose previous versions had provenance and whose new one doesn’t, because that’s what a stolen account publishing from somewhere else should look like. We’re writing it. I haven’t checked whether chalk or debug ever published with provenance, and if they didn’t, it wouldn’t have helped yesterday at all.

Two weeks ago, for comparison

This is the second of these in a fortnight. On August 26th, malicious versions of the Nx build tool went up, and that payload went looking for secrets on developers’ machines. One of the ways it did that was by running whatever AI coding CLIs were installed, with the flags that skip permission prompts, and asking them to search the filesystem for credentials. We don’t use Nx, so we spent that evening just checking. But every engineer on my team has Claude Code installed, and I’d never thought of it as something malware could pick up and use. It made me reread our guidance on where those flags are allowed, which is only inside the sandbox.

Hardware keys

The phishing worked because the second factor was a code you type. Npm supports hardware security keys, which won’t type a code into a page on the wrong domain, and Junon was using codes. So are we. We publish two small API-client packages to npm, and until yesterday the two accounts that can publish them used phone codes, like the rest of engineering has since 2020. Those two are on hardware keys as of this morning, and they’re moving to trusted publishing this week, so no one’s laptop can publish them at all.

The rest of engineering is still on phone codes for everything else, a split we made five years ago because support were the ones getting phished. After Monday I don’t think it holds up, so keys for every engineer are going into the Q1 plan, and I’m pricing them this week. It’s two keys per person, one to carry and one in a drawer, so a bit over forty keys.