Joe Armstrong and "let it crash"

Joe Armstrong died on Saturday, at 68. He started work on Erlang at Ericsson in 1986, with Robert Virding and Mike Williams, and spent the next thirty-odd years explaining, very patiently and usually with a joke, why the rest of us were handling errors wrong.

I’ve never shipped a line of Erlang. I’ve written toy programs, read Programming Erlang on a couple of long flights, and spent one weekend years ago convinced I was going to rewrite a chat server in it (I didn’t). But there are maybe five ideas from other people that changed how I think about building systems, and “let it crash” is one of them.

What it means

The phrase sounds reckless, and people who hear it secondhand usually take it that way. What he meant was close to the opposite. Erlang programs are made of a huge number of tiny, isolated processes. Not OS processes, Erlang ones, cheap enough that you can have hundreds of thousands of them, each with its own memory, talking only by sending messages. Because nothing is shared, when one of them hits something it doesn’t know how to handle, it can just die without corrupting anyone else’s state.

And something is always watching. Processes can be linked, so that when one dies the others hear about it, and the standard library builds that into supervisors: a process whose only job is to start some workers, notice when one dies, and restart it in a known good state. Supervisors supervise other supervisors, so you get a tree, and failures get handled at whatever level knows what to do about them. A worker crashes, its supervisor restarts it. If it keeps crashing too often in too short a window, the supervisor gives up and dies itself, and its parent restarts that whole subtree.

So the code in the worker doesn’t try to handle every possible error. It handles the ones it understands, which are part of the normal logic, and for everything else it crashes, loudly, with a log entry. Recovery lives somewhere else, in code whose whole job is recovery. His PhD thesis, which is from 2003 and is called “Making reliable distributed systems in the presence of software errors”, is the long version, and the title is the argument. You will have bugs. The question is what the system does when one fires.

What we stole

We don’t run Erlang. We run mostly C# and Ruby services on Linux and Windows boxes, and I’ve been writing C# long enough to have strong opinions about what it trains you to do. What it trains you to do, at least in the code I’ve reviewed over the years, is wrap things in try { ... } catch (Exception ex) { log.Error(ex); } and carry on. It feels responsible, and it’s usually the worst thing you can do, because after an unexpected exception you don’t actually know what state your objects are in, half an update might have happened, and now the process keeps running with that state and fails later in a way that’s ten times harder to debug.

Armstrong’s version, translated into the world we live in, has become a few rules on my team. Catch the exceptions you expect and can do something sensible about, a timeout you can retry, a validation failure you can report back. Let everything else take the request down, or the job, or if necessary the whole process. Make the process cheap to restart and stateless enough that restarting it is safe. Then put a supervisor over it, which for us is systemd or the Windows service recovery settings on the boxes, and the orchestrator in the container world, so a crash becomes a restart and a metric rather than an outage. And alert on the crash rate, not just on “is it up”, because something that restarts every thirty seconds is up and is also on fire.

It’s mostly about where the error handling lives. I’ve written before about checking the return values you think can’t fail, and I still believe that. The trick is to check them and then fail fast, rather than check them and try to limp on.

What we’ve never had is the cheap isolation underneath. Our unit of failure is an OS process or a container, which is heavyweight compared to an Erlang process, so we can’t “let it crash” at the granularity he could. Erlang lets a single phone call, or a single chat connection, die and restart on its own. We’re at the level of a whole service. That’s the part of his design I envy most, and the part that’s hardest to bolt onto a language that wasn’t built for it.

He was also very funny, which is rare in people who are this right about things. My favorite line of his is from Coders at Work, on object-oriented languages: “You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.” I’ve quoted it in more design reviews than I’d like to admit. If you’ve never seen one of his talks, there are lots of them on YouTube, and they’re all worth an hour.