"No moat": open models catch up

On my laptop right now there’s a 4GB file that’s a version of a model Meta released to researchers in February, quantized down to four bits per weight, running under llama.cpp at a few words a second. It’s not as good as ChatGPT. It’s a lot better than I thought anything running on an M1 MacBook Air would be in May 2023, and three months ago it didn’t exist.

That’s the argument of the memo SemiAnalysis published on the 4th, Google “We Have No Moat, And Neither Does OpenAI”, which they say is from a Google researcher, shared internally, and is one person’s opinion rather than the company’s. The core of it is a timeline. LLaMA announced on February 24th. The weights leaked on March 3rd. Within weeks people had it running on laptops and phones, fine-tuned on a laptop with LoRA (which trains a small low-rank update instead of the whole model), and instruction-tuned into Alpaca and then Vicuna, which its authors said comes close to ChatGPT quality. The memo’s conclusion is that “People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality,” and, a bit cheekily, that the clear winner of all this is Meta.

I think it’s half right.

The half I don’t buy is “comparable in quality”. Vicuna’s comparison used GPT-4 as the judge, asking it to score answers from each model, which is a clever cheap trick and not something I’d bet a product on. The local models I’ve tried are fluent and wrong much more often than GPT-4 is. In my own file of questions, the gap on the programming ones is not close. And LLaMA’s license is for non-commercial research, so the leaked weights and everything fine-tuned from them are off the table for a company like ours, whatever they can do. The open models with licenses we could actually use (Databricks’ Dolly 2.0, MosaicML’s MPT-7B, Stability’s StableLM) are newer and, so far, weaker.

The half I do buy is the speed. Every one of those gaps has been closing by the week, and the memo’s real point, buried in the middle, is that giant models are slow to iterate on and small ones are fast, so the people running lots of cheap experiments will keep catching up with the people running a few expensive ones.

What that means for us is mostly architectural. The internal tool we built on the OpenAI API last month, which sorts the bug reports our own staff file into categories and guesses which team owns each one (staff-written, with a filter that strips anything that looks like a customer identifier first, which is how it fits our new policy), talks to OpenAI directly from four places in the code. That was fine for a prototype. It’s the wrong shape if the model it’s calling might be replaceable in a year by something cheaper or something we host ourselves. So this week it got one small interface, with the prompt and the model name behind it, and a set of 200 hand-labeled reports we can run any model against to see if its answers are still good enough. That test set is the part I’d tell anyone to build first. Without it, “should we switch models” is a matter of opinion.

I don’t know who wins. If the memo is right, nobody makes much money on the model itself, and the money is in the products and the data around it. If it’s wrong, we’re renting from one of two or three companies for the foreseeable future. Either way I’d like to be able to swap the thing out in an afternoon.