TensorFlow, a practitioner's first look

I’m not a machine learning researcher, so take this for what it is. Google open-sourced TensorFlow on the 9th, and I’ve spent most of my evenings since then playing with it. I took Andrew Ng’s online ML class when it first ran, in the fall of 2011, and since then I’ve used what I learned there mostly through scikit-learn, for everyday things at work like a lead-scoring model and some text classification for support tickets.

TensorFlow is a library for building computation graphs over multi-dimensional arrays (the tensors) and then running them, on a CPU or a GPU. You describe the computation in Python, the graph gets executed by a C++ core, and the library can work out the derivative of any part of the graph with respect to any other part. That last bit is the thing that matters. It’s what Google uses internally for its neural network work, the successor to their DistBelief system, and it’s under the Apache 2.0 license.

Here’s the core of the beginner MNIST tutorial, a softmax regression on handwritten digits:

x = tf.placeholder("float", [None, 784])
W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x,W) + b)

y_ = tf.placeholder("float", [None,10])
cross_entropy = -tf.reduce_sum(y_*tf.log(y))
train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cross_entropy)

None of these lines compute anything. They build a graph. Then you start a Session, run train_step a thousand times with batches of 100 images, and the tutorial says you should end up at about 91% accuracy on the test set. I got 91.6%.

The line that got me is minimize(cross_entropy). In Ng’s class, the neural network assignment (exercise 4, in Octave) had you write the cost function and then backpropagation by hand, and then run a gradient check that compared your backprop against a numerical estimate, because you would get the backprop wrong. I did, several times. I remember an entire Saturday on an off-by-one in the bias term. The gradient checker said my gradients were off and gave me no idea where.

In TensorFlow you write the forward pass and the library differentiates it for you. That isn’t new, Theano has done automatic differentiation for years, but it’s the first time I’ve seen it packaged with documentation good enough that someone from my background can sit down and use it in an evening. I can see why the class made us do it by hand. I’d never have understood what backprop was without that Saturday. I also never want to do it again.

The other thing that feels very different from scikit-learn is the level you work at. In scikit-learn you pick a model, fit it, and tune hyperparameters. The model is a black box with a nice interface. In TensorFlow there’s no model, there’s just math, and you build the model out of matrix multiplies and nonlinearities. For logistic regression that’s more work than LogisticRegression().fit(X, y), by a lot. For anything that doesn’t fit one of scikit-learn’s boxes it’s much more flexible.

It’s a 0.5 release and it feels like one, which is fine, but a few things are worth knowing before you get excited.

The open-source version runs on one machine. The distributed version, the part that would let you train across a cluster the way Google does, isn’t in the release, and the Google people have said it’s coming. Since “Google-scale” is the whole pitch for a lot of people, that’s a big gap, at least for now.

GPU support is Linux only and needs CUDA 7.0, cuDNN, and a card with compute capability 3.5 or higher. That rules out the GPUs in Amazon’s g2 instances (they’re 3.0), which is what I’d have used. On the Mac it’s CPU only. On Windows it doesn’t install at all, which as someone who still spends a good part of every day on Windows I noticed. The binary packages are for Python 2.7.

And I’ve seen people saying that it’s slower than Torch and Caffe on the standard convnet benchmarks. I haven’t measured that myself, my models are too small to care, but if you’re doing serious image work it’s worth checking before you switch.

For the work I actually do, scikit-learn is still the right tool. Most of our problems are a few thousand rows of tabular data where a random forest or a regularized regression does fine, and nobody needs a GPU for that. I’m not going to rewrite a lead-scoring model in TensorFlow to feel modern.

But I think it’ll matter a lot. Google putting its internal tool out there with this level of polish means a lot of people who would never have touched Theano are going to start building neural networks, and the tutorials and Stack Overflow answers will pile up around whichever library they pick. I’d bet on this one, though Torch and Theano have a head start with researchers, and it’s early. I’m going to work through the convolutional tutorial next, since that’s the part where neural networks actually beat what I’d do with scikit-learn, and I’ll see whether the whole graph-then-session thing starts to feel natural or keeps feeling like writing a program to write a program.