fork() can fail

Rachel Kroll’s “fork() can fail” went around last week, and if you write anything that starts processes on Unix you should go read it. The gist: fork() returns -1 when it fails, which it will when you hit a process limit, and lots of code treats the return value as “0 means I’m the child, anything else is the child’s pid”. Store that -1 as a pid, and later on, when you go to clean up your child with kill(pid, SIGKILL), you’ve just called kill(-1, SIGKILL), which sends the signal to every process you have permission to signal. If you’re root, that’s pretty much the whole machine.

The HN thread was the usual argument about whose fault this is. One side says read the man page, the other says an API where an error value is also a valid and very destructive argument to another call is a bad API. I lean towards bad API, but it doesn’t matter much, because the API isn’t changing.

So I did the obvious thing and grepped our code for fork. Most of our stuff is in Ruby and Python, where fork raises an exception on failure, so a failed fork blows up loudly instead of handing you a -1. That’s the nice thing about exceptions for this kind of error, you can’t silently ignore one. But we have a small C helper on our worker boxes that supervises some long-running jobs, written years ago by someone who’s since left, and sure enough:

pid_t pid = fork();
if (pid == 0) {
    /* child */
    execv(path, argv);
    _exit(127);
}
children[n++] = pid;

No check for -1. And there’s a timeout path that kills anything in children that runs too long. It runs as a dedicated user, not root, so the damage would be limited to that user’s processes, which on those boxes is all the other workers. We’ve never hit it, as far as I know, but those boxes have hit the process limit at least once during a bad deploy, so it isn’t hypothetical. It’s fixed now, three lines.

What bugs me is how ordinary this is. I learned C on Turbo C under DOS, where malloc returning NULL wasn’t a theoretical case, you had 640K and you ran out regularly, so I check malloc. But I’ve never once written a test for fork failing, or close failing, or a write to a full disk, and I’d bet most people haven’t. These failures only happen when something else is already going wrong (too many processes, out of memory, out of disk), which is exactly when you least want your error handling to turn a bad day into a much worse one.

I don’t have a grand fix. We added a line to our code review checklist about unchecked return values on system calls, and I spent an afternoon running the helper under a ulimit -u low enough to make fork fail, which found one more problem (it logged the failure by shelling out to logger, which needs a fork too).