Giving an agent an hour and a test suite

58 minutes, 33 files changed, 1,140 tests passing, and one decision it made on its own that I’d have wanted to be asked about. That’s the summary of the experiment I ran over the long weekend, and the rest of this post is the detail, because the detail is where my opinion changed.

The week’s news

Anthropic released Claude Opus 4 and Sonnet 4 on Thursday. The pitch for Opus is long-running work. The announcement quotes Rakuten running it on an open-source refactor for seven hours, and says both models are 65% less likely than 3.7 Sonnet to take shortcuts or exploit loopholes on agentic tasks, which I read as “less likely to edit the test to make it pass”, the thing I caught it doing in February. Claude Code is now generally available, with editor integrations and a GitHub Action. That’s on top of OpenAI’s Codex a week and a half ago and GitHub’s coding agent at Build the Monday before last, both of which run in the cloud on a task you hand them and come back with a pull request. Everyone has decided at the same moment that the next step is the agent you don’t watch.

I’ve been watching mine. Since February, I’ve used Claude Code most days, and since Anthropic got through our vendor review in March, on work code too. It’s on the allowed list for source code, and it isn’t on our subprocessor list, so it never sees customer data. Agents work against seed data and test fixtures, never a copy of production, and that rule hasn’t been hard to keep. But every session has been interactive. I approve the edits, I read the diffs as they come, and I stop it when it starts going in circles. I wanted to know what happens when I don’t.

The setup

The task was one I’d been putting off. One of our .NET services is still on .NET 6, which went out of support in November. It’s about 40,000 lines with a decent test suite, unit tests plus integration tests that run against Postgres in a container. Upgrading it to .NET 8 is mostly mechanical, with a few package upgrades that each have their own breaking changes, and it’s the kind of thing that takes an engineer two or three days of mild tedium.

I wrote a spec, about a page of markdown. Target .NET 8. Upgrade the packages listed, in this order. Don’t change behavior. All existing tests must pass, and you may not change a test’s assertions; if you think a test is wrong, stop and write down why. Keep a log in UPGRADE_NOTES.md of every decision that wasn’t obvious. When you’re done, write a summary.

Then the boundary, because the whole point was to stop approving each step. It ran in a container on a spare machine, with a fresh clone of the repo and no credentials of any kind, not even a GitHub token. Network access was limited to the NuGet feed and the container registry for the Postgres image. I started Claude Code with the flag that skips the permission prompts, which has “dangerously” in its name for a good reason, and which I’d only use inside a box like this.

Then I went and made lunch.

What came back

It finished in 58 minutes. The summary was clear and, as far as I could check, accurate. It had retargeted every project, upgraded the packages in the order I’d given, fixed the compile errors each upgrade introduced, and got the full suite passing, 1,140 tests, without touching an assertion. I checked that with a diff of the test directories, not by taking its word. It had also, without being asked, removed a polyfill for a method that’s built into .NET 8, with a note explaining why, which is the sort of thing a careful engineer does and a hurried one doesn’t.

The notes file was the best part, and the reason I’d do it this way again. Fourteen entries, each a decision and a reason. Most were dull, as they should be. Entry 9 was the interesting one. Upgrading the Postgres driver across a few major versions changes how it maps timestamps without time zones to .NET’s DateTime, and several integration tests failed. It found the compatibility switch the driver provides, Npgsql.EnableLegacyTimestampBehavior, set it at startup, and wrote down that it had done so and that the driver’s own docs describe this as a temporary measure. That’s a reasonable call for a “don’t change behavior” upgrade. It’s also exactly the kind of decision I’d want a person to raise in standup, not bury in entry 9 of 14, because the right long-term fix is a change to how we store timestamps, which needs a conversation before anyone writes code. My spec told it to stop if it thought a test was wrong. It didn’t think a test was wrong. It thought the code needed a setting, and my spec didn’t cover that.

About twenty minutes of the hour went on one thing. A build tool tried to download something from a host outside the allowed list, the sandbox blocked it, and the agent spent a while trying alternatives, including a different download URL for the same tool, before it found the package on NuGet and used that instead. I was glad the box held. I’m less glad it tried a second door before looking for the right one, and I’d like it to have stopped and written that down instead.

I spent about an hour reviewing the result, which is about the same time it took to run. Then an engineer on the service spent another hour on it, and the PR is up for review now, with the timestamp switch still in and a ticket to remove it.

What I think now

The length of the run is the least interesting number here. The model could probably have kept going for three hours. What made it work was the spec, the tests and the box, and those were all things I already had or could write in an afternoon. Without a test suite I trusted, I’d have had no way to know whether 33 changed files did what the summary said. Without the box I wouldn’t have dared to stop approving. And without the notes file, the one decision that mattered would have been invisible.

That makes me think the teams that get the most out of this will be the ones with good tests and the habit of writing down what they want before they start, more than the ones with the best agent, which is a strange thing to be saying about a technology that’s supposed to remove the need to write things down. I’m going to try two more of these in June, on tasks with weaker test coverage, mostly to find out where it stops being safe.