What I ask in interviews instead of puzzles

In October a post called “Dear Startups: stop asking me math puzzles to figure out if I can code” did the rounds on Hacker News (it was on the front page on the 21st, with a few hundred comments), and I’ve had it open in a tab ever since because I keep meaning to write about it. We’re hiring again this quarter, so this seems like the time.

The complaint is in the title: being judged on math puzzles by people who never look at your code. A lot of the HN comments were the usual “but we need to know how you think”, and some of them had a point, but I mostly agreed with the post.

I should say that I’m the person who’s supposed to like these questions. My undergrad degree is in math, I like probability puzzles, and early on as an interviewer I asked plenty of them. The one I used most was the one about two eggs and a 100-floor building. I stopped when I noticed the thing it predicted best was whether the candidate had heard it before. People who knew it gave a slick answer in two minutes. People who didn’t spent twenty minutes under pressure, and some of the best engineers I’ve hired would have bombed it. Google’s head of people operations, Laszlo Bock, told the New York Times last summer that they’d looked at their own data and found brainteasers to be “a complete waste of time”, which made me feel better about having quietly dropped them.

So what do I ask instead? Here’s what our loop looks like right now.

Read some code

The first thing I do with every candidate, junior or senior, is hand them about 80 lines of real code and ask them what it does. It’s a cleaned-up function from our own codebase, with the names changed, and it has one real bug in it and a couple of things I’d flag in a code review. I tell them there’s a bug. I don’t tell them where.

Most of a programmer’s day is reading code, their own from six months ago or someone else’s, and almost no interview tests it. You learn an amazing amount in fifteen minutes. Some people start at the top and narrate line by line. Some skim the whole thing first and then go looking for the interesting part, which is what I do. Some find the bug in two minutes and miss the fact that the function does two unrelated things. I’m not grading on finding the bug (though most good people do), I’m watching how they build a model of something they didn’t write. Ever since the competency matrix post I’ve thought “code readability” is one of the better signals of level, and this is the most direct way I know to see it.

Debug something

The second exercise is a small program that’s broken, on a laptop, with whatever tools they want. Right now it’s a little web service whose tests pass but which returns the wrong totals on one endpoint. They get a debugger, a shell and an internet connection. Yes, they can search Stack Overflow. So can I, every day.

What I’m looking for is whether they form a hypothesis and test it, or just change things until it works. A surprising number of experienced people go straight to adding print statements in random places. That’s fine as a tool (I learned to debug with printf in Turbo C, and honestly I still do it), but there’s a difference between printing the thing you think is wrong and printing everything and hoping.

Build a small thing

The third is a short work sample, an hour or so, done in our office with one of us nearby to answer questions. It’s deliberately plain: take this CSV of orders, produce this report, here’s the edge case we care about. There’s no clever algorithm in it. What there is, is a lot of room to show taste, how they name things, whether they test it, what they do when the spec is ambiguous (it is, on purpose, in one place).

I used to do this as a take-home and I’ve mostly stopped. Take-homes are great for people with free evenings and bad for people with kids or a demanding current job, and you can’t tell how long someone spent on it. One hour on-site is fairer. If someone would rather do it at home, I’ll still offer that.

And then just talk

The rest is conversation. Tell me about the hardest bug you’ve fixed. What did you build at your last job that you’re proud of, and what would you change about it now? What’s a technical opinion you held strongly and then changed your mind about? That last one is my favorite question, because people who can’t answer it worry me more than people who can’t reverse a linked list.

I still ask some CS fundamentals, I’m not against them. If you’re going to work on our search code you need to know what a hash table costs. But I ask in the context of the work sample (“this is slow with a million rows, why?”) rather than as a quiz.

None of this is perfect. Work samples take more of our engineers’ time than a puzzle does, and reading code favors people who’ve worked in a language similar to ours, which I try to correct for by letting them pick from a couple of versions. I’ve also definitely hired someone who interviewed great on all of this and then struggled on the job, so it isn’t magic. But in about two years of doing it this way I’ve had far fewer “how did this person get through the loop?” moments than I had in the puzzle years.

If you’re interviewing people this month, try one thing: take a function from your own codebase, put a bug in it and ask the next candidate to read it out loud. Pick a function you’re slightly embarrassed by, those work best.