cat posts/2016-03-29-left-pad-and-the-11-lines.md

left-pad and the 11 lines that broke the build

· 4 min read · javascript tools dependencies

This is the whole thing, version 0.0.3, the one everybody’s builds were asking for:

module.exports = leftpad;

function leftpad (str, len, ch) {
  str = String(str);

  var i = -1;

  ch || (ch = ' ');
  len = len - str.length;


  while (++i < len) {
    str = ch + str;
  }

  return str;
}

Eleven lines if you don’t count the blank ones. It has a bug, too: leftpad(5, 3, 0) gives you " 5" and not "005", because 0 is falsy and gets replaced by a space. Which is a little funny, given what most people use left-pad for.

If you missed it: last Tuesday Azer Koçulu unpublished all 273 of his packages from npm, after a dispute where Kik (the messaging company) wanted the package name kik, he wouldn’t give it up, and npm ended up giving it to them. One of the 273 was left-pad. According to npm’s post-mortem, at about 2:30pm Pacific they started seeing hundreds of failures a minute from installs that couldn’t find it. Babel and React Native both depended on it, indirectly, and so did a lot of everything else. Someone published a replacement within ten minutes, but it was 1.0.0 and plenty of dependency chains asked for 0.0.3 specifically, so npm did something it had never done and republished the old version from a backup. All in, about two and a half hours.

David Haney’s post, NPM & left-pad: Have We Forgotten How To Program?, is the one that went around afterwards, and I agree with about half of it. The half I don’t agree with is the idea that the problem is people using tiny modules instead of writing their own. Writing your own left-pad is fine, and so is using someone else’s. The problem is having no idea that you’re using it.

Counting

So on Wednesday we counted. Our front-end build has 41 direct dependencies in package.json. npm ls --parseable | wc -l in that project says 1,140. I knew it was a lot. I didn’t know it was 28 times more than the list I’d actually reviewed. Of those, three are other copies of left-pad, at two different versions, pulled in by different things.

The more useful count was: if any one of these disappeared from the registry, which of our builds would break, and how would we find out? The honest answer was all of them, and from a red CI build with an error message about a package nobody on the team had ever heard of. We got lucky last week only because nobody happened to deploy in those two and a half hours.

What we changed

Every other ecosystem learned this years ago.

  1. We commit a shrinkwrap file. npm shrinkwrap writes out the exact version of every package in the tree. We’d been relying on ^ ranges, so two builds a day apart could get different code, which was its own problem before left-pad and nobody had fixed it. Ruby has had Gemfile.lock for years, PHP has composer.lock, and we’ve had shrinkwrap available this whole time and just didn’t use it.
  2. We set up a caching proxy so every package we’ve ever installed stays on our own box. If something vanishes from npm, our builds don’t notice. Sinopia does this and was about an hour to set up.
  3. Builds produce an artifact and deploys use it. That was already true for our backend and not for the front end, which was still running npm install on the build box per deploy. Now the build makes a tarball, and a deploy never touches the network for dependencies.
  4. When shrinkwrap changes in a PR, a reviewer at least skims the list of what’s new. It won’t catch anything clever, but it’ll catch “why did this PR add 60 packages?”.

We talked about checking node_modules into the repo, and decided against it, mostly because of native modules that compile per platform. If you deploy to the same OS you build on, it’s a reasonable option and a lot of people do it.

npm says it will make it harder to unpublish a package that other packages depend on, which is the right call, if overdue. For comparison, NuGet, which I’ve used since it showed up in the .NET world, has never let you delete a package from nuget.org, you can only unlist it, so it stops showing up in search but anything that already depends on it still restores. I remember being annoyed by that once when I’d pushed a broken version. I’m not annoyed by it anymore.

The bigger thing for me is that the registry is someone else’s server, run by a company, with policies, and we had our entire build pipeline depending on it being up and unchanged, without ever having made that decision on purpose.