cat posts/2021-07-06-github-copilot-first-week.md

GitHub Copilot, first week with an AI pair programmer

· 4 min read · ai tools programming

The first thing I typed after installing it was this:

def parse_duration(s: str) -> int:
    """Parse strings like '1h30m', '45s' or '2h' into seconds."""

and before I’d hit Enter after the docstring, Copilot had filled in a regex, a loop over the matches, a dictionary mapping h, m and s to multipliers, and a return. It was correct. I tried '1h30m' and '90s' and an empty string, and the only thing I’d have done differently is raise on garbage instead of returning 0. I sat back and said something out loud that I won’t repeat here.

That was the high point of the week, and I want to be honest that it came in the first minute, because a lot of what’s being written about Copilot right now is based on that first minute.

What it is

GitHub announced it on June 29 as a technical preview, an extension for VS Code only for now, with a waitlist. It’s built on a new OpenAI model called Codex, which GitHub says does better than GPT-3 at code because it was trained on a much larger amount of public source code. As you type, it sends the surrounding code to the service and shows a gray suggestion inline, anything from the rest of a line to a whole function, and you press Tab to take it. GitHub says it works especially well for Python, JavaScript, TypeScript, Ruby and Go. I got in off the waitlist a couple of days after the announcement, so this is about five days of use, not a week.

I only used it on a side project, a small Python service with a Ruby script or two around it, and not on anything from work. It sends your code to GitHub to generate suggestions, and I haven’t read the terms carefully enough yet to be comfortable pointing it at code that isn’t mine. I’d recommend anyone else check that before installing it on a work laptop.

The tally

For about two days I kept a rough count in a text file, which is the kind of thing I do instead of enjoying things. Of the multi-line suggestions I actually read (I ignored most single-token ones), about a third I accepted as they were, maybe a fifth I accepted and then fixed, and the rest I rejected. That’s not a scientific number, but it matched how it felt. It’s right often enough that you keep reading the suggestions, and wrong often enough that you have to.

Where it was good:

  • Boilerplate that follows a pattern already in the file. Write one test case and it writes the next four, often with sensible edge cases.
  • Glue code against very popular libraries, like requests, argparse or the standard library’s datetime, where it’s clearly seen a million examples.
  • Small, well-specified functions with a clear docstring, like the one above.
  • Converting things: a dict to a dataclass, a SQL query to the equivalent ORM call, a list of JSON fields to a schema.

Where it wasn’t:

  • Anything specific to my code. It doesn’t know what’s in other files unless they’re open, so it invents method names that sound like they should exist on my classes.
  • Library APIs that changed in the last couple of years. It happily wrote an older form of a call that’s since been deprecated, presumably because there’s more of the old version on GitHub.
  • Anything where being almost right is the problem. It wrote me a date-range check with < where I needed <=, a retry loop that retried on every exception including KeyboardInterrupt, and a regex that looked fine and had a nested quantifier I wouldn’t put in front of user input (I wrote about what that can do a couple of years ago).

That last category is what I’d worry about with less experienced engineers. When I write a bug, I usually have a vague feeling that something’s off, which is what makes me look again. Copilot’s bugs come with no such feeling. They’re fluent and confident and formatted beautifully, and reading code carefully is a harder skill than writing it. A junior engineer who accepts suggestions faster than they can review them is going to ship things they don’t understand, and I’m not sure how a code reviewer would even tell.

The licensing question

The other thing that’s been all over Twitter this week is what it was trained on. GitHub’s FAQ says that about 0.1% of the time, a suggestion may contain snippets verbatim from the training set, mostly when there’s very little context. One clip doing the rounds shows it reproducing the Quake III fast inverse square root, famous comment included. Quake’s code is GPL. If a tool hands you a few lines of GPL code without telling you, whose problem is that? GitHub’s position seems to be that training on public code is fair use, and it’s working on a way to flag suggestions that match the training set. I’m not a lawyer, and I don’t think anyone actually knows the answer yet, including the lawyers. For now I’d treat anything long and suspiciously polished that it produces the way I’d treat a snippet from a random blog: find out where it came from before it goes in.

Am I keeping it?

I’m going to keep using it on the side project. It’s made the tedious 30% of that code faster, and it’s a surprisingly good way to discover a library function I didn’t know existed, since it’ll use one and I’ll go look it up. Writing a clear comment first and letting it suggest the code is a nice way to work, and it pushes me to write the comment, which is no bad thing.

But “AI pair programmer” oversells it. A pair programmer knows the codebase and argues with me. This is autocomplete trained on the whole of GitHub, and it’s very good autocomplete. I’d want it in Visual Studio for C#, where I spend a lot of my time and where it isn’t available yet, and I’d want a much better answer on licensing before I suggested it to the team.