GPT-4 and my moving skepticism

I keep a text file of things I’ve asked these models and whether the answer was right. It started with Copilot in 2021, carried on through ChatGPT in December, and as of this morning it has 187 lines. Since last Tuesday, when GPT-4 showed up in ChatGPT Plus, the ratio of right to wrong in that file has changed enough that I’ve had to admit my position has moved, so this is me writing down where it’s moved to before I forget where it started.

Where it started: in 2020 I wrote that GPT-3’s arithmetic falling apart at four digits was the warning label, and that a model that’s right 80% of the time is, for most programming purposes, a model that’s wrong. In 2021 I said Copilot was very good autocomplete and that “AI pair programmer” oversold it. I still think both of those were fair at the time. GPT-3.5, the model behind ChatGPT, was more useful than either, but it was mostly useful for things where I could check the answer faster than I could produce it, and it was wrong in a confident, fluent way often enough that I treated everything it said like a first draft from someone who’d skimmed the docs.

OpenAI announced GPT-4 on the 14th. The headline numbers in the announcement are exam scores: on a simulated bar exam it lands around the top 10% of test takers, where GPT-3.5 was around the bottom 10%. I don’t care much about the bar exam. The technical report runs to about a hundred pages and contains, by its own account, no details about the architecture, the model size, the hardware, the training compute or the data, citing competition and safety. I care about that a bit more, since it means everything I know about this model comes from using it. It can take images as input, but that isn’t available to the public yet. The API has a waitlist, and when you do get in it’s $0.03 per thousand prompt tokens and $0.06 per thousand completion tokens for the 8K context, double that for the 32K one, which is a lot more than the ChatGPT model. Microsoft also confirmed the same day that the new Bing has been running on it all along, which explains a few things about February.

So here’s what a week of real use looks like, from the file.

The first thing that changed is SQL. I gave it the schemas of four tables from our reporting database (column names and types, no data) and asked for a query I’d been putting off, a cohort retention report with a couple of awkward edge cases around accounts that downgrade and come back. GPT-3.5 had got this wrong in December in a way that took me twenty minutes to find. GPT-4 got it right the first time, including a LEFT JOIN I’d have forgotten, and when I asked why it had used a window function where I’d have used a subquery, the explanation was correct. I checked the output against a hand count for two cohorts and both matched.

The second is reading other people’s code. We have a PowerShell script from 2016 that rotates logs on our last few Windows boxes, and nobody who wrote it is still here. I pasted it in and asked what it did and where it would break. The summary was accurate, and it found a real bug: it compares file dates as month/day/year strings, so every January, December’s logs sort as newer than the current month’s and never get rotated. That would explain a disk-space alert we get every winter and had blamed on something else. I’d read that script before and missed it.

The third is the stuff around code. Writing the first draft of a migration plan, turning a rambling incident timeline from Slack into a clean one, explaining a git error message to me that I’ve seen a hundred times and never understood. None of these are hard. All of them used to take me twenty minutes, and now take five, and over a week that adds up to more time than I’d have guessed.

Where it’s still wrong: it invented a parameter for an AWS CLI command that doesn’t exist, very plausibly, in exactly the style of the real ones. It told me with complete confidence that a particular R function had an argument that was removed several versions ago (that one I only caught because I’d hit the same deprecation during my certificate). And when I gave it a fiddly date-range bug and asked it to fix it, it produced a fix that passed my example and broke the boundary case, which is the same kind of mistake Copilot made in my first week with it. The difference is that the wrong answers are now a small minority of the file instead of a large one. Over the week, 41 of the 53 programming questions I logged came back right, or right enough that I only had to fix the formatting. That’s one person’s file and nowhere near a benchmark, and I picked the questions, but in December the same kind of week came out around half and half.

The other thing that’s changed is that it says “I’m not sure” more, or at least it qualifies things, which GPT-3.5 almost never did. It’s not consistent about it, and the qualified answers aren’t always the wrong ones. But I’ve had a few answers this week where it told me which part it was less confident in, and it was right about that too, which is a strange experience.

So where has my skepticism moved to? I’m no longer skeptical that this is useful for everyday programming work. It is, it’s saving me real time, and I’ve stopped thinking of it as a toy. I’m still skeptical of anything where I can’t check the answer, of anyone who tells me it understands the code (I don’t know what it’s doing, and the report won’t say), and of the idea that a junior engineer can lean on it the way I can. Most of the value I’m getting comes from being able to tell, quickly, when it’s wrong. Someone who can’t tell is getting a different tool, and possibly a dangerous one.

There’s also a question I’ve been ignoring all week, which is that every one of those 53 questions went to a third party’s servers, including the four table schemas and a PowerShell script with internal hostnames in it. I’ve been treating my own use as an experiment. Three people on the team have asked me this week whether they’re allowed to use it at work, and I’ve been saying “be sensible” while I think about it, which isn’t an answer. We need an actual policy, and I suspect writing it will be harder than any of the questions in the file.