2023: how AI changed my team's year

In January the only AI tool in regular use was Copilot, on personal subscriptions the company reimbursed, for about half the engineers. By December every engineer had it, about half the company used ChatGPT in some form most days, we had a feature in production built on GPT-3.5, and I’d written more words about language models on this blog than about anything else. That’s a big change for one year, and I’ve been trying to work out over the holidays how much of it was real.

So this is a list of specific things that changed, with the before and after where I have them. Most of the numbers are rough. We didn’t measure carefully at the start, which is the first thing I’d do differently.

Things that changed

Small scripts and one-off queries

This is the biggest change and the hardest to measure. Before: someone in support or finance needs a number, files a request, and an engineer writes a SQL query against the reporting database when they get to it, usually a day or two later. We were averaging about a dozen of those a week in the spring. After: we gave the support leads and someone on the finance team read-only access to the reporting database, a page describing the main tables, and permission to use GPT-4 through our API account to write the queries (the model sees the schema and the question, never the results), with an engineer reviewing anything that’s going to be used for a decision. By November it was down to three or four requests a week, and those were the hard ones. Nobody’s job was the SQL queries, but they were taking a real slice of two people’s weeks.

Ticket summaries

The feature I wrote about during the OpenAI weekend. Before: an agent picking up a ticket with a long thread read the whole thing. After: a summary at the top. Our support lead measured handle time on threads with more than 20 messages for a month before and a month after, and the median went from about 11 minutes to about 7. Short tickets didn’t change, which is what you’d expect. (The OpenAI story resolved itself within a few days, with Altman back as CEO, and the fallback work we did that week stayed in. The eval set is the most useful thing to come out of it.)

Tests and boilerplate

Copilot for Business became generally available in February, and we moved every engineer onto it in March at $19 a seat. Where it clearly helped was the code nobody enjoys writing: test cases after the first one, request validation, mapping one shape of object to another, migration scaffolding. I asked the team in November, anonymously, whether they’d object if we cancelled it, and 16 of our 19 engineers (we were nearer 30 before the layoff in November 2022) said yes, most of them strongly. Two said they’d switched it off months ago and forgotten, and one didn’t mind either way. Compared with my first week with it in 2021, it’s better at using context from other open files, and it’s still, as I said then, wrong in ways that look exactly like right.

Postmortems

Before: whoever led the incident spent half a day turning the Slack channel into a timeline. After: we export the channel, run it through a script that strips customer names and account IDs, have GPT-4 draft the timeline with timestamps, and the incident lead corrects it, which takes about an hour. The analysis part, why it happened and what we’re changing, is still written by a person, and I’ve asked that it stay that way. When I tried letting the model draft that section, it produced something that read like a postmortem and contained no information.

Reading unfamiliar code

Pasting a 300-line function from an old service into GPT-4 and asking what it does is now normal for us. Under the AI policy we wrote in April our code doesn’t go into the consumer ChatGPT, so someone built a small internal chat page on our API account, and it’s the most-used internal tool we made this year. This is the use I’d keep if I could only keep one. It’s not always right about the details, but it’s a fast way to get a map before you start reading.

Hiring

Our work sample, the one-hour exercise we run over video, stopped telling us much once we’d opened it up to AI tools in April. By the summer, about half the solutions had the same structure and the same comments, and the follow-up conversation was doing all the work. We changed the exercise. Instead of writing something from a blank file, the candidate now extends a small piece of working code and talks us through it as they go, which is closer to what I described back in 2014 anyway, and they can still use whatever tools they like. Watching how someone uses Copilot turns out to be informative.

Things that didn’t change

Design work. Every hard decision this year, the database migration in the summer, the cloud bill work, the tenancy changes, was people at a whiteboard and in documents. I tried using GPT-4 as a sounding board on a couple of design docs, and it’s good at listing considerations and bad at telling you which one matters.

Debugging the hard bugs. The ones that took days this year took days. AI helped with the easy part of those, writing a quick script to replay some traffic, and nothing with the part where you figure out what’s actually going on.

Code review got harder. PRs are bigger on average, because generating code is cheap, and reviewing it isn’t. I don’t have a solution to this yet beyond asking people to keep PRs small, which is the same thing I’ve been asking for ten years.

The bug we wrote twice

In October I wrote about a Unicode truncation bug, a Substring that cut an emoji in half in a customer’s name. We fixed four of the six places it appeared. In November an engineer added a new preview field, Copilot suggested the same Substring(0, n) pattern, and it went through review with two approvals. A test that someone had added after the October bug failed in CI. The model learned that pattern from code like ours, and I expect it to keep suggesting it for as long as code like ours exists.

Where I’ve ended up

I came into the year skeptical, moved a long way in March when GPT-4 came out, and ended it somewhere in between: these tools are a real and lasting change to how my team works, mostly by taking the friction out of small tasks, and they haven’t touched the hard parts of the job yet. Whether that second part stays true is the question I’m least sure about for 2024. The jump from GPT-3.5 to GPT-4 was large enough that I don’t trust my sense of what next year’s models won’t be able to do.

The thing I want to fix first in January is measurement. I have one decent before-and-after number this year, the handle time on long tickets, and it’s there only because our support lead thought to measure it. Next year, anything we roll out gets a baseline first.