GPT-5 and choosing models on evals, not launches

143 cases. GPT-5 did better than our current model on 13 of them, worse on 6, and cost about a third less per case. That was our support eval set on Friday morning. It’s a fine result. It’s not the result you’d have predicted from Thursday’s livestream, and it’s not the one you’d have predicted from Twitter on Saturday either, when half my feed had decided GPT-5 was a disappointment because it couldn’t count something or other.

The launch

OpenAI released GPT-5 on Thursday. In ChatGPT it’s really a system, a fast model and a “thinking” model with a router in front that decides which one answers you, and it replaced everything else in the model picker for most people. In the API it’s three sizes, gpt-5, gpt-5-mini and gpt-5-nano, with the big one at $1.25 per million input tokens and $10 per million output, which is cheaper on input than GPT-4o. They report 74.9% on SWE-bench Verified, the coding benchmark everybody quotes, against 69.1% for o3.

The livestream had a chart where the bar for 69.1% was drawn the same height as the one for 30.8%, and Sam Altman called it a “mega chart screwup” afterwards. People had a lot of fun with that. And the removal of GPT-4o from ChatGPT went badly enough, with people who’d got attached to its personality, that by the weekend OpenAI had said paying users could have it back.

It was a big week before that, too. Anthropic shipped Claude Opus 4.1 on Tuesday, and OpenAI released two open-weight models, gpt-oss, the same day. I remember when a week like this would have had me rewriting plans. This time I mostly felt tired.

Why I don’t trust launch day

I’ve now been through enough of these to notice a pattern in my own reactions. GPT-4 in 2023 was a real jump and I moved a long way that week. Most launches since have been smaller than the excitement around them, and a few have been bigger in ways that only showed up after a month of using them, which is the opposite problem. Benchmarks don’t help much. SWE-bench measures fixing issues in a set of Python repos, and our code is Ruby and C# with a TypeScript front end, and our hardest problems aren’t issues anyone has written down.

So I’ve stopped deciding on launch day. What decides is the eval set.

That’s the set we started building during the OpenAI weekend in 2023, 60 real support tickets with summaries a support lead wrote and marked as good. It’s grown since. It has the summaries, and since the drafted-reply pilot in the spring it has 83 reply cases too, including the twelve bad drafts from the spring that invented a policy or a menu, which every model has to get right. Each case has a rubric, and the scoring is a mix of a model doing the first pass and a person reviewing anything that changed. Running a new model on it takes a morning, most of it the person.

What GPT-5 did

Because OpenAI is already on our subprocessor list, and has been since 2023, we could run it on real tickets the day it came out. No new vendor review, no customer email. That matters more than it sounds. It’s the reason a launch from OpenAI gets tested the next morning and a launch from anyone else gets tested on public data and my side projects.

The summaries were slightly better. Of the 60, the reviewer preferred GPT-5’s version on 9, preferred the current one on 4, and called the rest a wash. Two of the four were cases where GPT-5 left out a detail the customer had mentioned once, early on, that turned out to matter.

The drafts were the interesting part. On the 83 reply cases, there were 4 fixes, where the current model got something wrong and GPT-5 didn’t, and 2 regressions. One of the regressions was a draft that offered a trial extension “on request”, which is one of the nine from April and exactly the kind of sentence our citation rule is supposed to catch. The citation highlighting did catch it, so it wouldn’t have reached a customer. It still counts as a regression.

Latency was worse with reasoning on. At the default reasoning effort, the median draft took about twice as long as today. At minimal it was about the same as today, and the quality was close to the default on our cases, which I didn’t expect. Cost at minimal came out about a third lower per case, mostly because of the input price, since our prompts are long and our outputs short.

What we’ll do

Probably switch the summaries in September, after another run with the prompt tuned a little. The drafts, I don’t know yet. Two regressions out of 83 isn’t much, but one of them was the exact failure we built the rule to stop, and I’d rather understand why before swapping the model under the pilot. That’s going to take a week of someone’s attention, which is the real cost of a new model, much more than the API bill.

None of this is a review of GPT-5. It might be the best model in the world at things we don’t do. What I’ve noticed is that once you have an eval set, launch day stops being an event. A new model is a row in a spreadsheet, with a date, and a number that’s slightly better or slightly worse than the row above it, and most weeks that’s all it is. I find that a relief, honestly. I’d been reading the launches like sports results for two years.