Hiring engineers when agents do the typing

“Will it count against me if I use an agent?” A candidate asked me that at the start of her work sample on Wednesday, and she’s the third this month. The answer has been no since 2023, when we started letting people use whatever tools they like in the work sample. That people still ask says something about other companies’ interviews, and it made me look harder at ours.

The loop is still recognizably the one I described in 2014: a code reading exercise, a debugging exercise, a one-hour work sample and a conversation, over video now. It was designed to look like the job. The job has changed a lot in the last eighteen months, and the loop had only changed at the edges.

What we replaced

The code reading exercise used to be a few hundred lines of real-looking code to explain. But the reading our engineers do all day now is reviewing changes an agent made and deciding whether they’re right. So since April, the reading exercise is a pull request. It’s a realistic change to a small service, written by an agent from a real-looking ticket, with a test suite that passes. It has two problems in it that we know about. One is a test whose assertion was quietly loosened, and the other is a new helper function that duplicates one already in the codebase with slightly different behavior, which are two of the problems I found most often when I went through our own reviews this spring. The candidate gets thirty minutes and the ticket, and we ask them to review it as if a colleague had sent it.

Of the eleven senior candidates who’ve done it so far, four found both problems, five found one, and two approved it with style comments. The ones who found the duplicate were, every time, the ones who went looking outside the diff, which is the habit I care about most.

What we watch for in the work sample

The work sample itself is the same hour, a small working codebase and a feature to add. What changed is what we watch, because most candidates now drive an agent for a good part of it.

The strong ones spend the first five or ten minutes reading the code themselves, then give the agent a specific task with a definition of done. They read its diffs as they arrive, and they stop it when it’s heading somewhere wrong, usually saying why. The weaker ones type the feature description into the agent in one go and then wait, and when it’s done, they run the tests and say it works. Both groups often end with working code. Only one group can tell us what the code does and where it might break, and that’s the part we ask about in the last fifteen minutes.

The best agent driver

One candidate this month was the best agent driver I’ve seen. Her prompts were short and exact, she caught two of the agent’s mistakes before they landed, and she finished the feature with twenty minutes to spare. Then, in the review exercise, she approved the pull request with three comments about naming. I asked her about it afterwards and she said, honestly, that she’d assumed the tests were the check, because in her current job they are.

The skill I’m surest we need is the review, and she didn’t show it. But she might be one good code review culture away from having it, and her other hour was the strongest of anyone’s. We’ve asked her back for a second conversation, which I almost never do, mostly so I can find out whether I’m the one being unfair.