Book review: Superintelligence

I read Nick Bostrom’s Superintelligence: Paths, Dangers, Strategies over the last couple of weeks, partly because it’s been getting a lot of attention this fall. It came out in the US in September, and in August Elon Musk tweeted that it was worth reading and that AI is “potentially more dangerous than nukes”, which is the kind of thing that sells a philosophy book. Bostrom runs the Future of Humanity Institute at Oxford, and the book is a careful, slightly airless academic argument that machine superintelligence, if we build it, could be the last thing we build, and not in a good way.

I want to review it as a programmer, because most of the reactions I’ve seen are either from philosophers who grant the premise or from people dismissing the whole thing as science fiction, and I don’t fit either. I’ve written software for twenty years. I took Andrew Ng’s online machine learning class when he first put it up, back in late 2011, and did all the exercises, so I’ve implemented gradient descent and backprop and logistic regression by hand, badly, in Octave. That’s not expertise, but it does mean that when Bostrom talks about an AI I’m not picturing a robot, I’m picturing the gap between the code I actually write and the thing he’s describing, and that gap is the whole problem I have with the book.

The argument

Bostrom’s case, stripped down, is a chain. One: human-level machine intelligence is possible in principle, because brains are physical and there’s no reason the thing they do can’t be done in silicon. Two: once you have a machine around human level, it may be able to improve itself, and a thing that’s good at building intelligence improving its own intelligence could go fast, maybe very fast, an “intelligence explosion”. Three: there’s no law that says a superintelligence would share human goals; intelligence and goals are separate (his “orthogonality thesis”), so you could get something extremely capable that wants something we’d find pointless or horrifying. Four: a superintelligence pursuing almost any goal would, as a side effect, want resources and want to not be turned off, because those help with nearly any goal (his “instrumental convergence”). Put those together and you get his worry: a system much smarter than us, pursuing a goal we specified carelessly, that treats us as either raw material or an obstacle.

His standard illustration is the paperclip maximizer: an AI told to make paperclips, which is good enough at it to turn the whole planet, us included, into paperclips, not out of malice but because that’s literally what we asked for and it was better at it than we expected. The serious version of this is the “value loading problem”: how do you specify what you actually want to a system that will take you exactly at your word and is far better than you at finding the cheapest reading of it. Anyone who’s written a reward function, or for that matter a SQL query that did precisely what they typed and not what they meant, feels the shape of this immediately.

What’s good

The value-specification chapters are the best part, and they’re the part a working engineer should read even if they bounce off the rest. Bostrom is very good on the ways a literal optimizer defeats the thing you thought you were asking for, and a small version of that is something I run into at work most weeks. Every one of us has written a metric that got gamed, an A/B test that optimized for the wrong click, a cache that was technically correct and operationally a disaster. Scale that up and give it real capability and his concern stops sounding silly. The orthogonality point also lands: there’s a comfortable assumption that smart implies nice, and nothing in my experience of either software or people supports it.

He’s also honest, mostly, about how uncertain all of this is. The book is full of hedges, and the timelines he quotes (he surveys AI researchers on when human-level AI might arrive, and the answers are all over the place, with a lot of weight decades out) come with big error bars. He is not saying this is coming soon. He’s saying that if it comes, the control problem is hard enough that we should be thinking about it well in advance, and that’s a reasonable thing to say.

Where I get off the train

My problem is with the second link in the chain, the leap from “human-level” to “explosion”, and more basically with how far today’s systems are from the first link at all.

The AI I actually work near is machine learning, and the recent excitement, the stuff that’s made image recognition suddenly good over the last two or three years, is deep neural networks trained on a lot of data and a lot of GPUs. It’s genuinely impressive and I don’t want to undersell it. But it is a very long way from anything with goals. It’s curve-fitting, extraordinary curve-fitting, but a classifier that tells cats from dogs has no more desire to acquire resources than a spreadsheet does. Bostrom knows this; he’s careful to say he’s not claiming current techniques scale to general intelligence. But the book spends almost all its energy on what happens after you have a general agent, and almost none on whether the path from here to there exists, and the gap between “we can fit functions well” and “we have an agent that recursively self-improves” is, as far as I can tell, not an engineering gap you cross by adding GPUs. It might be one conceptual breakthrough away, or ten, or it might be that general intelligence doesn’t “explode” at all because it runs into the same grind of diminishing returns that every other hard problem does. Bostrom treats the explosion as a live enough possibility to build the book on. I think it’s possible, and I don’t think anyone, him included, knows enough to put a number on it.

There’s also a flavor to the argument that bugs me, and it’s the same flavor as a lot of pure math: once you accept the definitions, the conclusions follow airtight, and you’re left unsure whether you’ve learned something about the world or just about the definitions. Orthogonality and instrumental convergence are clean, almost theorem-like. Reality is where my code meets a network timeout and a disk that’s 97% full, and I’ve never seen cleanliness like that survive contact with it.

Should you read it

If you build software, read the middle third, the value-loading and control chapters, and treat them as a very sharp essay about specifying goals to things that take you literally. That part is real and it’s useful now, today, for anyone writing an optimizer or a metric, long before any of the science-fiction stuff. The far-future scenario-spinning I’d take with more salt than the author does, not because it’s impossible, but because the first step, actually getting to a general agent, is the step we understand least and the one the book spends least time on.

I came out of it more careful about reward functions and no more worried about the end of the world than I went in, which may be exactly the wrong reaction, and I’m honest enough to admit I can’t prove it isn’t. Ask me in twenty years, if anyone’s still asking.