Copilot, a year in

The test file had seven test methods in it by the time I’d written one. I’d written the first, for a small C# class that parses our customers’ CSV imports, named the second ParsesQuotedFieldWithEmbeddedComma, and stopped typing, and Copilot filled in the body, and the next five method names, and their bodies. Six of the seven were tests I’d have written. The seventh asserted that an empty line should produce an empty record, which was wrong for our format, and which I only caught because I read it.

That’s still the best one-paragraph description I have of using it, fifteen months after my first week with the technical preview. Then it was a side project in Python and Ruby, and I wasn’t willing to point it at work code. Now it’s in Visual Studio on my work machine, and I use it every day.

What changed

GitHub made Copilot generally available in June, at $10 a month or $100 a year, free for students and for maintainers of popular open source projects. The Visual Studio extension that I wanted in that first post had already arrived in preview, so C# was covered. There’s still no plan for companies, so you can’t buy it for a team or manage it centrally. It’s a personal subscription that each engineer signs up for.

So we wrote a short policy in July. Anyone can use it on our code. The company reimburses the subscription. The setting that blocks suggestions matching public code has to be on, and we asked people to opt out of snippet collection where the settings allowed it. It’s not allowed in two repos, the billing service and the one with our authentication code, mostly because I don’t want to have that conversation with an auditor yet. About half the engineering team signed up in the first month. A few people tried it for a week and turned it off, and I think their reasons were as good as the reasons of the people who kept it.

Where it saves time

Tests, more than anything. Once a test file has two or three tests in it, Copilot is very good at the next one, and it’s good at the dull edge cases I’d skip when I’m tired: the empty list, the null, the off-by-one at the end of a range. I still have to read every one, as in that CSV example, but reading a test is faster than writing one.

Mapping code. C# has a lot of it, DTOs to domain objects and back, and it’s the most tedious code I write. Copilot writes it correctly most of the time, because there’s nothing to get creative about.

And the bits of an API I don’t remember. I write a comment like // retry with exponential backoff, max 5 attempts, using Polly and it writes something close to the Polly syntax, which I never remember. It’s a faster way of looking something up than a browser tab, as long as I check it against the docs when it matters.

GitHub published a study last month where 95 developers wrote an HTTP server in JavaScript, half of them with Copilot, and the Copilot group finished in an average of 1 hour 11 minutes against 2 hours 41 minutes for the others, about 55% faster. I believe it, for that task. A small greenfield server in a popular language, with a clear spec, is exactly where it’s strongest. Most of my week looks nothing like that, and my own guess for the time it saves me is a lot smaller, maybe an hour or two a week, almost all of it in tests and glue.

Where it costs time

The cost is the reading. Every suggestion is a little interruption. I’m thinking about what I want, a grey block appears, and now I’m thinking about what it wants and whether that’s the same thing. Most of the time the answer comes quickly. Sometimes I read a twelve-line suggestion carefully, decide it’s subtly wrong, delete it, and write the thing I was going to write, and that was a minute and a train of thought gone. For anything hard I now turn it off. There’s a keyboard shortcut for that, and I use it more than I expected to.

Then there’s reviewing other people’s code. Copilot’s mistakes look exactly like correct code, nicely formatted and named sensibly, and I wrote about that worry fifteen months ago. Here’s what I’ve actually seen since. Twice this year I’ve found a bug in review that the author told me came from a suggestion they hadn’t read closely, both times in code that compared dates. Neither was a junior engineer. It’s the experienced people who are fast enough to accept a suggestion without reading it, I think, and the less experienced ones who read every line, because they don’t trust themselves yet.

It also makes me lazier about things I used to know. I noticed in August that I’d stopped remembering the exact string.Join overloads, because I don’t type them any more. I’m not sure that matters, but I noticed.

The licensing question, again

In my first post I said nobody knew whether Copilot reproducing licensed code was anyone’s problem. That’s still true, but it’s closer to being tested. Last week Tim Davis, a professor at Texas A&M, posted a comparison showing Copilot producing what looked like his own sparse-matrix transpose code, which is LGPL, with no license or attribution attached. A couple of days later Matthew Butterick, who’s a lawyer and a programmer, announced that he and a law firm are investigating a possible class action against GitHub over Copilot. Someone from GitHub replied to Davis that the code was different, though similar, which is a weak defense if you look at the two side by side.

I’m not a lawyer and I won’t guess where this goes. What I can say is why we turned the public-code filter on. It’s a crude check, and Davis’s example suggests it doesn’t catch everything, but for our code it turns “we can’t know” into “we tried”. Most of what Copilot writes for me is too specific to our codebase to be anybody else’s code. The risk is in the long, polished, general-purpose functions, and those are the ones I already treat like a snippet from a random blog.

A year in

I’d keep paying for it with my own money. That’s the real test, I think, and it passes. But I’d describe it now the same way I described it then, as very good autocomplete, and the people on our team who get the most from it are the ones who treat it that way. The ones who treat it as a pair programmer end up reviewing its work without meaning to, and they’re not always good at it, and neither am I.