Llama 2 on my laptop

7.3GB. That’s the size of the file sitting in my Downloads folder right now, a 4-bit quantized copy of Llama 2 13B Chat, and it answers questions on the M1 MacBook Air I bought in 2020, with the Wi-Fi turned off. I’ve been using computers for thirty years and that still felt like a magic trick the first time.

Meta released Llama 2 on Tuesday, together with Microsoft, in three sizes (7B, 13B and 70B parameters), each as a base model and a chat model tuned with RLHF. According to the paper they were trained on 2 trillion tokens, about 40% more than the first LLaMA, with a 4,096-token context. There’s a 34B model in the paper too, but they’ve held it back, saying they didn’t have enough time to red-team it.

The difference from the first LLaMA is the license. Back in March you had to be a researcher to get the weights legitimately, and most people got them from a torrent anyway. This time you fill in a form on Meta’s site, get an email with a download link, and you’re allowed to use it commercially.

Getting it running

I used llama.cpp, which has had GPU support on Apple Silicon through Metal since June. The 7B and 13B models are the same architecture as the original LLaMA, so they just worked. The 70B uses a different attention arrangement (grouped-query attention) and, as of yesterday, llama.cpp didn’t handle it yet. It wouldn’t fit in 16GB of RAM anyway, even quantized, so I didn’t try.

Converting Meta’s weights yourself is a couple of commands. I did it once for the 7B to see how it worked, and then downloaded someone else’s pre-quantized GGML files for the 13B, which is what most people seem to be doing. That’s a bit of trust you’re handing to a stranger on Hugging Face, and I’m not entirely comfortable with it for anything beyond playing around.

On my machine the 7B chat model generates around a dozen tokens a second and the 13B roughly half that, which is slower than reading speed but not painfully so. The fan question doesn’t come up, because there isn’t one. The Air gets warm and stays warm.

Is it any good?

The 13B is about where GPT-3.5 was when ChatGPT first came out, on the things I tried, and clearly behind GPT-4. I gave it the same handful of tasks I’ve been using to compare models since spring: explain a gnarly SQL query from our codebase (with the table names changed), write a regex for our log format, summarize a long and rambly support thread, and turn a bullet list into a short release note. It did the release note and the summary well. The SQL explanation was mostly right and confidently wrong about one join. The regex was wrong in a way that took me five minutes to spot, which is the same complaint I’ve had about every model.

The chat tuning is cautious to the point of comedy. I asked the 7B how to kill all the zombie processes on a Linux box and got a paragraph about how violence isn’t the answer before it got anywhere near ps. (Strictly speaking you can’t kill a zombie, you have to deal with its parent, and it didn’t mention that either.) I’ve seen several people post similar refusals this week. The base model doesn’t have that problem, and I expect less jumpy fine-tunes of it to show up within weeks, if not days.

Why I care

Our AI usage policy, which we wrote in April, says customer data doesn’t go into third-party chat tools. That rules out the most useful thing I’d like a model for at work, which is reading a pile of support tickets and telling me what people are actually complaining about. A model that runs on a laptop, or on one of our own servers, changes that conversation. Nothing leaves the machine. Whether the 13B is good enough at the job is something I’ll find out over the next few weeks, on a sample I’m allowed to use.

It also matters as a hedge. Everything we’ve built with LLMs so far calls one company’s API, and I’d like to know that if the price or the terms change, there’s something we could run ourselves that’s 80% as good.

The license, read properly

Meta calls it open source. It isn’t, by the usual definition, and I think it’s worth being precise about it rather than arguing over the word. A few things in the Llama 2 Community License that people should actually read:

• If your products had more than 700 million monthly active users on the release date, you need a separate license from Meta. That’s aimed at a handful of companies and won’t affect anyone I know.
• There’s an acceptable use policy you agree to, which is a list of things you can’t use it for.
• You can’t use Llama 2 or its outputs to improve any other large language model, other than Llama 2 and its derivatives. That one is a real restriction, and the sort of thing a lawyer should look at before anyone builds a training pipeline on it.

For a startup like ours, none of that is a blocker. It’s still a license written by one company that can presumably write a different one for Llama 3, which is the same dependency I just said I wanted to hedge against, only one step removed.

I’m going to try the ticket summarizing on the 13B next week and see how it compares to GPT-3.5 through our API account. GPT-3.5 can’t see real customer tickets (OpenAI hasn’t been through vendor review for customer data, and the policy doesn’t allow “anonymized” exports either, which I agree with), so our support team is writing up fifty realistic ones from our own test accounts, and both models get the same set. If it’s close, that’s a much more interesting result to me than any benchmark in the paper.