KV caches and why context is expensive

The model was 8.5GB. The laptop was the old 16GB MacBook Air I keep for experiments. I set the context length to 128K tokens, because the model supports it and I wanted to feed it a long log file, and the server process died before it had read a single token of the log, having tried to allocate about twice as much memory as the machine had.

Nothing about the model had changed. What changed was the KV cache, and since it’s behind more of the day-to-day behavior of agents and APIs than anything else I can think of, I wanted to draw it out.

What’s being cached

I walked through attention in the transformer post a few years ago. The short version is that at every layer, each token is turned into three vectors. Its query asks “what am I looking for?”, its key says “what do I contain?”, and its value is what it hands over if someone attends to it. To produce the next token, the newest token’s query is compared with the keys of every earlier token, and the result decides how much of each earlier token’s value to mix in.

During generation, the model produces one token at a time, and each new token needs the keys and values of everything before it, at every layer. Those don’t change. Token 40’s key at layer 12 is the same whether you’re generating token 41 or token 4,100, because a model like this only looks backwards. So instead of recomputing them on every step, the server computes each token’s keys and values once and keeps them. That’s the KV cache. Without it, generating each token would mean re-running the whole prompt.

How big it gets

Each token needs a key and a value, at every layer, for every key-value head, each a vector of the head dimension. So the cache per token is

2 × layers × KV heads × head dimension × bytes per number

Llama 3.1 8B is a good example because its configuration is public and well known. It has 32 layers, 8 key-value heads, and a head dimension of 128. At 16-bit precision, that’s 2 × 32 × 8 × 128 × 2 = 131,072 bytes, exactly 128KiB, for every token of context.

That doesn’t sound like much until you multiply. At 8,000 tokens it’s 1GiB. At the model’s full 128K context (131,072 tokens), it’s 16GiB, which is about twice the size of the model’s own weights at 8-bit. That’s my laptop crash. The weights were fine. The server tried to reserve room for a full-length cache up front, and 8.5 plus 16 doesn’t go into 16. At 8K context the same setup runs happily.

The 70B version has 80 layers and the same 8 KV heads, so it’s 320KiB per token and 40GiB for a full context, for one conversation. A server holding many long conversations for many users is mostly a machine for storing KV caches, with a model attached.

And those numbers are after a trick that already cut them by 4x. Llama 3.1 has 32 query heads but only 8 key-value heads, an arrangement called grouped-query attention, where groups of four query heads share one set of keys and values. With a full set of 32, the 8B model would need 512KiB per token, and 64GiB for a full context.

Why long context costs more than you think

Memory is half of it. The other half is that every new token has to read the whole cache, at every layer, so the work per token grows with the length of the context. Generating token 100,001 means attending over 100,000 cached positions, and each of those reads comes out of the same memory bandwidth that’s already the bottleneck for reading the weights. Long contexts make every token slower, as well as making the conversation bigger.

That’s also why prompt caching exists and why it’s priced the way it is. If you send an API the same 50,000-token prefix twice, the system prompt and tool definitions and a codebase summary, say, the provider can keep the KV cache for that prefix around and skip computing it the second time. Anthropic charges a tenth of the normal input price for tokens read from the cache, and a bit more than normal (1.25x for the five-minute version) to write it. Those prices make sense once you know what’s being stored. A cache read is the provider handing you back work it already did. A cache write is them agreeing to hold gigabytes of your keys and values in fast memory for a while, which isn’t free.

The catch is that a cache only works for an exact prefix. Change one token near the start of the prompt and everything after it has to be recomputed, because every later key and value depended on it. I learned that the expensive way last year. A small tool of mine put the current date and time at the top of its system prompt, so no two requests ever shared a prefix and the cache never hit. Moving the timestamp to the end of the prompt cut the bill for that tool by more than half.

What people do about it

There’s a lot of work on making the cache smaller. Grouped-query attention is the most common. DeepSeek-V2, in 2024, went further with what they call multi-head latent attention, which stores a compressed latent vector per token and reconstructs the keys and values from it, cutting the cache by a large factor. Some models use sliding-window attention on some layers, where those layers only look back a few thousand tokens and so only cache that many. And you can quantize the cache itself. llama.cpp lets you store keys and values at 8-bit, which halves the 16GiB, and on my laptop I couldn’t tell the difference in output on the tasks I tried.

On the serving side, vLLM’s PagedAttention (2023) manages the cache the way an operating system manages memory, in fixed-size pages allocated as a conversation grows, instead of reserving a full-length block up front. That’s exactly the mistake my laptop server made, and fixing it is one of the reasons serving these models got cheaper.

What it means for agents

A coding agent’s context only grows during a session. Every file it reads, every test run and every diff gets appended, and stays, and each new step pays to attend over all of it. Most of the habits that make agents work better are, underneath, ways of keeping that cache short and stable.

Starting a fresh session instead of dragging an old one along is one. Compaction, where the agent summarizes the conversation so far and starts over from the summary, is another, and the reason it seems to make the agent dumber for a minute is that a summary throws away detail that was in the cache. Sub-agents that do their searching in their own context and send back a few paragraphs are the same idea again. And keeping the instructions file at the top of a session short and unchanging is good for a reason I didn’t appreciate until I worked through the numbers above: it’s the prefix, so it’s what gets cached, and every edit to it invalidates the cache for everything after it.

I used to think of context length as a feature you either had or didn’t, like a bigger hard disk. It’s closer to RAM in a DOS-era PC, where you could always get more but paid for every kilobyte, and my old Turbo C habits about not wasting it turn out to be useful again.