Local models are good now

My morning email has been written by a model on my own laptop for three weeks, and I didn’t notice the switch. That’s a small Ruby app I’ve had for years, which pulls my RSS feeds overnight and sends me a digest, and since last year it’s put a two-line summary under each article. Those summaries used to come from an API. In April I pointed the same code at a server on my laptop instead, changed the model name, and the digests kept arriving looking exactly the same. I only remembered I’d done it because the API bill for that key went to zero.

So Cyrus Lopez’s Local AI Needs to be the Norm, which he wrote in December and which Hacker News found yesterday, landed with me. His argument is that when an app sends user content to a cloud model for a simple feature, the developer signs up for a pile of obligations and failure modes they didn’t need: privacy and retention questions, provider outages, rate limits, billing, a server of their own in the middle. And most of those features, like summarizing, classifying, extracting and rewriting, don’t need a frontier model. His example is his own iOS news app, which summarizes articles on the device with Apple’s on-device model, so no text leaves the phone and there’s no AI provider account at all. That’s almost exactly what my digest app does, except mine runs on a laptop and his runs on everyone’s phone, which is the harder and more useful version.

What changed

Three years ago I ran Llama 2 13B on a MacBook Air and called it roughly GPT-3.5 on a good day. In early 2025 it was the small DeepSeek-R1 distills, and the 14B was the biggest that fit in 16GB. Both were fun to play with, and I didn’t use either for anything real.

This spring is different, and I think it’s because of two things. The first is mixture-of-experts models at sizes a laptop can hold. Qwen3.6 35B-A3B, which came out in April, has 35 billion parameters but only uses about 3 billion of them for each token, so it needs the memory of a 35B model (a bit over 20GB at 4-bit) and runs at roughly the speed of a 3B one. On my current MacBook Pro it’s fast enough that I don’t wait for it. The second is that the open models are just better. Google’s Gemma 4, released at the start of April under Apache 2.0, and the Qwen models are the first local models I’ve used where I stopped mentally adding “for a local model” to my opinion of the answers.

My setup is short. llama.cpp’s server, running in the background, with an OpenAI-compatible API on localhost. The Qwen MoE as the default, and a Gemma 4 model for when I want a second opinion. Anything that can talk to that API works with it, which turned out to include the digest app, a couple of scripts that tag notes, and a couple of agent harnesses that accept a custom endpoint, which I’ve tried and will come back to.

What I use them for

The digest. Tagging and sorting my notes, which are personal and which I’d rather not send anywhere. Turning a long log file into “these three errors, in this order”, which they’re good at and which often has hostnames and customer IDs in it. Anything on a plane. And a lot of “what’s the flag for…” questions, where the answer is a line and a round trip to a frontier model is overkill.

At work, the policy question is interesting. A model running on an engineer’s laptop doesn’t send anything to anyone, so it isn’t a new vendor and doesn’t go through the vendor review that every AI provider has. That’s real, and it’s part of why I’m keen. It doesn’t make the laptop a place customer data is allowed to be, though, and we’ve had to say so explicitly, because “it’s local, so it’s fine” was the first thing two people said when I raised it.

Where I think he’s wrong

Lopez is careful to say he’s not against cloud AI, and that local models lag the big ones. Where I’d push back is on how big that lag is for the work I actually do most, which is agents writing code.

I tried the Qwen model in an agent harness for a few evenings on my own projects. For a single, well-specified change in one file it was fine. For anything that needed twenty steps, reading several files, running tests and adjusting, it lost the thread far sooner than the hosted models do, and it was much worse at noticing that it had lost it. Some of that is the model and some is that I can’t give it a long context on a laptop without running out of memory (I wrote about why last week). For agentic coding, the gap between what I can run and what I can rent is still large, and I don’t expect it to close this year.

For features inside an app, which is what his essay is actually about, I think he’s right, and I’d go further. If a feature is summarizing, classifying or rewriting the user’s own data, sending it to a cloud model should be the thing you have to justify. A year ago I’d have said the quality gap justified it for most features. For most of them now, it doesn’t.