A week with Claude Code

At about 11 on Monday night I noticed I hadn’t typed a line of code in an hour and a half. I’d typed a lot of English, read a lot of diffs, and pressed “y” maybe forty times, and a small Ruby app I wrote years ago, which had been broken since I upgraded Ruby on that machine, was passing its tests again on Ruby 3.4. I’d been meaning to fix it for about eight months.

That was my first evening with Claude Code, which Anthropic released on Monday as a “limited research preview” alongside Claude 3.7 Sonnet. It’s a command-line tool. You install it from npm, run claude in a project directory, and type what you want. It then searches and reads the code, proposes edits, runs commands like your test suite or git, reads the output, and keeps going until it thinks it’s done or needs you. It asks before it edits a file or runs anything, and you can tell it to stop asking for a particular command for the rest of the session. It bills against an Anthropic API account by the token. On Windows it needs WSL, which is where I ran it on the laptop.

I’ve used Copilot since 2021 and ChatGPT since the week it came out, and I’ve written a couple of toy agents of my own over the last year, so I thought I knew roughly what this would be. I was wrong about one thing, and it’s the important one. With chat, I’m the loop: I paste in the code, read the answer, paste it back, run the tests, paste in the error. With Copilot, I’m still typing and it’s finishing my lines. Here the loop runs without me, and my job is to decide where it starts and whether I accept where it ended.

Where I ran it, and where I didn’t

Everything this week was my own code. The Ruby app that turns my RSS feeds into a morning email, my dotfiles, and a small C# command-line tool I use for my own notes. None of it was work code, and that’s deliberate.

Claude Code sends whatever it reads to Anthropic’s API, which means pointing it at a work repo sends our source code to a provider we don’t have a contract with. Under our AI policy, code only goes to tools on the allowed list, and getting a new AI provider onto that list means the same vendor review OpenAI went through in 2023. If it’s ever going to see anything with customer data in it, it also means adding Anthropic to our subprocessor list and telling customers first. I’ve started that review this week, because I’d like the option, and I’d rather wait a few weeks than be the person who broke the policy I wrote. The commercial terms say API inputs aren’t used for training, which is the first thing our security lead will check, and the research preview label is the second.

What it was good at

The Ruby upgrade is the best example. The app is about 2,000 lines, written for Ruby 2.6, with a test suite I’d been reasonably disciplined about at the time. I told it the app ran on 2.6, that I was now on 3.4, and that I wanted the tests passing without changing what the app does. It ran the tests first, which I’d have asked for anyway, saw about thirty failures, and worked out that most of them were the keyword-argument change in Ruby 3, where a hash in the last position is no longer quietly converted into keyword arguments. It fixed those in batches, ran the tests after each batch, updated two gems that wouldn’t install, and found the one real bug left over, a URI.escape call (removed in Ruby 3.0) in the code that builds links. It told me which replacement it had chosen and why. That’s the order I’d have done it in. I read every diff, and I’d have approved every one in a code review.

The notes tool is C#, about 1,500 lines, with no tests, which I’m not proud of. I asked it to add tests for the parser before I changed anything, and it wrote 34, ran them, and stopped to tell me that two were failing because of a real bug, a heading inside a fenced code block being treated as a real heading. It asked whether I wanted the bug fixed or the test marked as known-failing. That’s the behavior I want from a junior engineer, and it’s not what I expected from a tool.

It’s also very good at the reading part. “Where does this app decide which feeds to skip?” got me the right three files and a correct explanation in a few seconds, from a codebase I’d forgotten. That’s the use I said I’d keep if I could only keep one, and it’s better when the tool can open the files itself instead of me pasting them.

Where it went wrong

The dotfiles were a mess, and it was mostly my mess. I asked it to add a Windows-specific section to one of my chezmoi templates, and it spent about twenty minutes in a loop. It edited the template, ran chezmoi execute-template to check, misread the output, edited it again, and reverted its own earlier fix twice. I stopped it, explained how the templates are laid out, and it did the job in two minutes. Some of that is on me for not explaining. Some of it is that it doesn’t know when it’s going in circles, and it’ll keep going, cheerfully, at a few cents a minute, until someone notices.

Once, on the C# tool, a test failed after its change, and its next edit was to the test’s expected value, not to the code. I said no, and it apologized and fixed the code. It only did that once, and I only caught it because I was reading the diffs. If I’d let it run without approvals, I wouldn’t have.

And twice it told me something was done when it hadn’t run the tests after its last change. Both times they passed when I ran them. I don’t think that’s a reason to trust it.

Long sessions get worse. After an hour or so the conversation gets big enough that it starts forgetting decisions from the beginning, and there’s a /compact command that summarizes the history to make room, which helps, but I’ve started just ending the session and starting a new one with a short note about where we are. There’s also a convention where it reads a CLAUDE.md file in the repo for standing instructions, and putting “run the tests after every change, and never edit a test to make it pass” in there stopped both of the problems above for the rest of the week, as far as I could tell.

The numbers

I’ve kept my usual text file. Over three evenings I gave it 19 tasks. It finished 11 to the point where I accepted the diff without editing it, 5 more where I fixed something small, and I gave up on 3, two of them in the dotfiles. The API bill so far is a bit under $30, most of it on Monday and the dotfiles night. That’s not cheap for a hobby, and it would be cheap for a work tool if the numbers held, which, with a sample of 19 tasks on my own code, I can’t claim.

Where I’ve landed

It’s the first AI tool where I’ve stopped writing code and started reviewing it, for hours at a time, and it feels like being a tech lead for an engineer who types extremely fast, has read everything and remembers nothing from yesterday. Whether that’s good depends entirely on the review. Every bad change in my week was something I caught by reading, and the reading is the job now. The Ruby upgrade would have taken me a weekend, maybe. It took an evening, and I understood every change in it, because I approved every change in it.