Code review when agents write the code

How many pull requests did you review last week? I asked the team that in our Monday meeting, and the median answer was eleven. When I asked the same thing in early 2024, it was about six. Nobody’s PRs are bigger, because we halved the size guideline last year. There are just a lot more of them, and almost every one was mostly written by an agent, with a person steering.

That’s changed what review is for, and our checklist hadn’t caught up. It was written for human mistakes, like a forgotten null check or a variable named data2, and those barely happen now. What happens instead is a different set of mistakes, and I spent a couple of weeks in March reading other people’s reviews to work out what they were. This is what I’ve ended up telling people.

Read the tests before the code

The first thing I look at in an agent-written PR is the diff to the test files, on its own. Agents are very good at making tests pass, and occasionally the way they make a test pass is by changing what it expects. I caught Claude Code doing this on my own code in its first week, and the instructions we give agents tell them not to, and they mostly don’t. Mostly. In March I found two PRs where an assertion had been loosened, one where an exact match became a “contains”, and one where a test had been marked skipped with a comment saying it was flaky. It wasn’t flaky. It was correct.

So the question for the tests is whether any existing expectation got weaker, and if it did, whether the PR description says why. A new test is easy to review. A changed old one is where the bodies are.

Does it belong here?

The code itself is usually locally good, with clear names and error handling where you’d want it. It compiles, the tests pass, and if you read just the diff, it looks like a senior engineer wrote it.

Where it goes wrong is the system around the diff. The most common problem I found in March was duplication. An agent needed to format a date for an invoice, didn’t find our existing helper (or found it and decided a new one was cleaner), and wrote a third date-formatting function. Each one is fine. Three of them, with slightly different time zone handling, is a bug report from a customer in Australia in about six months. Close behind that is pattern drift, where the agent solves a problem the way most code on the internet would, not the way we decided to do it in this codebase years ago, like retrying an HTTP call inline when we have a retry policy for exactly that.

A diff tool can’t show you any of this, because the problem is code that isn’t in the diff. The reviewer has to know the system. That’s the strongest argument I’ve got for the slow, unglamorous kind of seniority, the person who’s been here four years and knows there’s already a function for that. I’ve asked reviewers to write “is there an existing way to do this?” in their heads at the top of every PR.

Look for what’s missing

The other big category is absence. An agent implements what it was asked for, and it implements it well, and it doesn’t do the thing nobody mentioned, like the audit log entry every other admin action writes, or the migration that has to run before the deploy. A human engineer who’d been in the planning meeting would have known. The agent knows the ticket.

We’ve started fixing this upstream, by putting more of that context into the tickets and into the instructions files in each repo, which has helped. But a reviewer reading a PR should have the ticket open next to it and ask what a person who’d been in the room would have added.

The author still owns it

Since last summer every PR description says how it was made, by hand, agent with approvals, or agent unattended. That’s worked. The addition this spring is one line: the author names the one or two places they’re least sure about. We got a lot of “I’m confident in all of it” in the first week. What I want is the real answer, the query they didn’t profile or the edge case they reasoned about without testing. That tells the reviewer where to spend their twenty minutes, and it’s a quick way to find out whether the author actually read the code, because you can’t name the weakest part of something you haven’t read.

What to stop doing

Stop reviewing style. We have formatters and linters, and agents follow them better than people do. Any review comment about spacing or naming conventions is time not spent on the above.

And stop rewriting. When a reviewer sees a problem in an agent-written PR, the temptation is to fix it in a suggestion block, because it’s quick. I’d rather they describe the problem and let the author, and their agent, fix it, because then the author learns what was wrong, and the repo’s instructions file sometimes gets a new line out of it so the next PR doesn’t have the same problem. That loop is a lot of how our agents have got better at our codebase over the last year. It only works if review comments explain things.

We also run a reviewer agent on every PR now, before a person sees it, mostly for the mechanical stuff. It catches real things. For the first few weeks it also wrote a tidy summary at the top of the PR saying the change looked good, and I watched a reviewer read that summary and approve. So it doesn’t give verdicts any more. Its instructions now say to list findings and nothing else, and the findings go in a comment below the description, where a person has to scroll past the author’s notes to reach them.