Picking a model per task

Three frontier model releases in fifteen days. GPT-5.1 on the 12th, Gemini 3 on the 18th, along with a new agentic editor from Google called Antigravity, and Claude Opus 4.5 on Monday, at a third of the price of the Opus before it. If you add the smaller ones (GPT-5.1-Codex-Max last week, Claude Haiku 4.5 last month, Sonnet 4.5 at the end of September), I’ve had a new model to think about roughly every four days since autumn started.

It’s Thanksgiving, and I’ve got a quiet morning before the day starts, so I went through my text file of agent tasks to see which models I actually use for what. The answer is that I’ve drifted, without really deciding, into a set of rules of thumb. Writing them down seemed like a better use of the morning than reading a fourth set of benchmark charts.

The rule that comes first

At work, a model is a vendor before it’s a model. OpenAI went through our vendor review in 2023 and is on our subprocessor list, so it can see customer data, with notice to customers that we’ve already given. Anthropic went through review this spring and is allowed for source code, but it isn’t on the subprocessor list, so it doesn’t see customer data. Google isn’t approved for any of it. So any question that starts “which model is best for summarizing tickets?” has really been answered before it’s asked: it’s one of OpenAI’s, and the only choice is which. Gemini 3 might be better at tickets. I have no way to find out without real tickets, and I’m not going to start a vendor review and a round of customer emails on the strength of a launch week.

That sounds like a constraint, and it is, but I’ve come round to it being a useful one. It makes the eval set the only argument that counts, and it takes away most of the temptation to switch every fortnight.

The rules

These are for me, mostly for coding with agents. The team has its own habits, and I haven’t tried to standardize them beyond the vendor rule.

Use the mid-size model by default. For everyday agent work on our code that’s been Claude Sonnet 4.5 since it came out. It’s fast enough that I stay in the loop, and good enough that most tasks don’t need more.
Use the big model for the long or tangled jobs. Upgrades, anything I’ll leave running unattended, and bugs where the first attempt failed. That used to be Opus 4.1, which I used sparingly because it was expensive. Opus 4.5 is $5 per million input tokens and $25 output, against $15 and $75 for 4.1, so as of Tuesday I’m trying it as my default for a couple of weeks to see if the line between rule 1 and rule 2 should move.
Use the smallest model that passes the eval for anything high-volume. Classifying, extracting, tagging. On our support evals the small models are within a point or two of the big ones on summaries and noticeably worse on drafted replies, so summaries get the small one and drafts don’t.
Get a second opinion from a different company’s model when I’m stuck. When Claude and I have gone around a bug twice, I paste the problem (code only, never data) into GPT-5.1 through our API account. I don’t think it’s a better model. It’s wrong in different places, though, and about one time in three that’s enough to get me unstuck.
Turn reasoning up only when the task needs planning. The thinking settings on these models make a real difference on design questions and almost none on “rename this and fix the call sites”, where they just make me wait.
Don’t switch models in the middle of a task. Each one has its own habits about how it structures code and what it assumes, and a half-finished change from one, picked up by another, has twice been worse than either would have done alone.

Gemini 3

I’ve used Gemini 3 Pro on my own projects since last Wednesday, through the CLI and a bit through Antigravity, and on a couple of open-source codebases where nothing is private. It’s very good. On a front-end task in my RSS app, building a settings page from a description, it was the best of the three I tried, by a margin I noticed without measuring. On a Ruby refactor it was about the same as Sonnet. It handles very long context better than I expected, and I gave it the whole of one open-source project to ask questions about, which worked.

None of that changes the first rule. If Gemini 3 is still clearly ahead on something we need in a few months, that’s when it’s worth asking our security lead for the time. We’ve done it twice, and it’s weeks of work each time, and with a Google contract there’s also the question of which Google, since we already found a Google Cloud account we’d forgotten about once.

What the text file says

Since September, about 70% of my agent tasks ran on the mid-size model, 25% on the big one, and the rest were small-model experiments or second opinions. My acceptance rate, the share of tasks where I took the result with at most small edits, was a little higher on the big model, even though it got the harder tasks, so the real gap is probably bigger than it looks. I don’t have enough tasks to say by how much, and I’m not going to pretend I do.

The thing the numbers don’t show is that the differences between the three companies’ best models, on my work, are smaller than the differences between a good spec and a bad one. I can make any of them look brilliant or useless by how I describe the task. Which is probably why everyone’s favorite model is different, and why mine will probably have changed again by the time I write the year-end post.