A week with ChatGPT

[Fact]
public void November2022Has21BusinessDays()
{
    Assert.Equal(21, Calendar.BusinessDaysInMonth(2022, 11));
}

ChatGPT wrote that test for me last Wednesday, along with nine others, for a small function I’d pasted in that counts the weekdays in a month. The other nine were fine. This one is wrong. November 2022 starts on a Tuesday and has 22 weekdays, and my function, which knows nothing about holidays, correctly returns 22. The test fails. What’s interesting is that 21 isn’t a random number. It’s the number of US business days in November once you take out Thanksgiving. It answered a slightly different question from the one I’d asked, put the answer in a test with a confident name, and didn’t mention any of that.

That’s my week and a half with it, more or less.

What it is

OpenAI released ChatGPT on November 30 as a free “research preview”. It’s a chat interface over a model fine-tuned from their GPT-3.5 series, trained with human feedback to follow instructions and to hold a conversation. Sam Altman tweeted on the 5th that it had passed a million users, and I believe it, because every engineer I know has been pasting screenshots into chat ever since. The same day, Stack Overflow put up a temporary policy banning answers generated by it, because people were posting them by the hundreds, and they look right far more often than they are right.

OpenAI’s announcement is fairly honest about this. The first item under limitations says it “sometimes writes plausible-sounding but incorrect or nonsensical answers.” That sentence is worth keeping in mind for every single answer.

I’d read the GPT-3 paper in 2020 and I’ve used Copilot every day this year, so I thought I knew roughly what to expect. The model underneath isn’t a huge jump from what I’d seen. The interface is. Being able to say “no, the function should also handle February in leap years” and have it revise its last answer is a different experience from prompt engineering, and a lot more people will use it because of that.

Where it helped

Unit tests, with the caveat above. I gave it four small functions over the week, two in C# and two in Ruby, and asked for tests. The tests were well organized, covered edge cases I might have skipped, and used the frameworks we actually use once I told it which ones. About one test in ten was wrong, always in the expected value and never in the structure. That’s still faster than writing them, because the wrong ones fail and you look.

Explaining things. I pasted in a 140-character regex from our email parsing code that nobody on the team has fully understood since the person who wrote it left, and asked what it did. The explanation was excellent, piece by piece, and it pointed out one alternative branch that can never match, which turned out to be true. I checked every claim against a regex tester, and it was right on all of them. I’ve used it the same way for a dense SQL query and a bash script, and it was good at both.

First drafts of things I don’t enjoy writing. A migration script skeleton. A Dockerfile for a small Ruby service. An email to a vendor that needed to be firm and polite. I rewrote all of them, but starting from something is quicker than starting from nothing.

Where it didn’t

Writing regexes, as opposed to explaining them. I asked for a regex to validate an IPv4 address and got ^(\d{1,3}\.){3}\d{1,3}$, with a paragraph explaining that it checks each octet is between 0 and 255. It doesn’t. It happily matches 999.999.999.999. When I said so, it apologized and gave me a correct pattern with the usual 25[0-5]|2[0-4]\d|... alternation, which made it worse in a way, because it clearly knew how to do it and had told me the first one was fine.

Facts about specific software. I asked how to cap the number of idle connections in Postgres, and it told me to set max_idle_connections in postgresql.conf, with a suggested value and a sentence about restarting the server. Postgres has no such setting. What it wrote is a very reasonable guess at what the setting would be called if it existed, and that’s the problem in one example. When it doesn’t know, it doesn’t say so. It produces the most plausible text, and plausible text about software looks exactly like documentation.

It also doesn’t know about anything recent. It says its training data goes to 2021, so questions about this year’s versions of anything get last year’s answers, also confidently.

Over the week I kept a rough count, which I seem to do with every new tool. Of the answers I checked properly, around one in five had something wrong in it that mattered. That number is about as unscientific as they come, since I was choosing what to ask, but it held up across the week.

Where I’ve landed

I’m going to keep using it, the way I’d use a well-read colleague who has never once said “I don’t know”. It’s good for explanations and first drafts, as long as I’m already able to judge whether the answer is right. And it’s dangerous for anything where I can’t check, because the wrong answers aren’t marked in any way. They’re written in the same clear, friendly paragraphs as the right ones.

That’s also why I’m more worried about it than about Copilot. Copilot’s mistakes are in code, and code gets run, reviewed and tested. ChatGPT’s mistakes are in prose, about how things work, and nobody runs a unit test on an explanation. I’ve asked our team to treat it like a search result from a site they’ve never heard of. Several of them are already using it more than I am, and I expect to write about this again in a few months, probably with a different opinion.