Speculative decoding, visually

About 17GB. That’s roughly how much a 32-billion-parameter model weighs at 4-bit quantization, once you add the bits of it that stay at higher precision. To produce one token, the machine has to read essentially all of it out of memory and through the processor. If your memory bandwidth is 400GB/s, which is in the range of a decent laptop with unified memory, that’s a ceiling of a bit over 20 tokens a second, before the processor has done anything clever, and no amount of extra compute moves it.

That number is the reason speculative decoding exists. It’s a trick I like a lot, because it gets a 2x or 3x speedup out of an observation about hardware without changing a single output token (the proof of that is about half a page of probability). I’ve been running it at home for a couple of weeks and I wanted to draw it out properly.

Why generation is slow

When a model reads your prompt (the “prefill”), it processes all the prompt tokens in parallel. That’s a big matrix multiply, the hardware is busy, and it goes fast. When it generates, it can’t do that. Token 101 depends on token 100, so each step is a full forward pass through the whole network for a single position. The arithmetic for one position is small. The weights it needs are not. So during generation the processor spends most of its time waiting for memory, and its arithmetic units sit mostly idle.

Here’s the observation. Running the model on five positions at once costs almost the same as running it on one, because the expensive part, reading the weights, happens once either way. The extra arithmetic for four more positions is close to free on hardware that was idling anyway. If we only knew what the next five tokens were going to be, we could check all five in one pass.

We don’t know them. But a much smaller model can guess.

Draft, then verify

You run two models. The target is the big one whose output you actually want. The draft is a small, fast model that uses the same tokenizer, usually from the same family. Each round goes like this:

1. The draft model generates γ tokens the normal way, one at a time. It’s small, so this is cheap. (γ is usually somewhere between 3 and 8.)
2. The target model runs once over the prompt plus all γ draft tokens, in parallel, like a tiny prefill. That one pass gives it its own prediction at every one of those positions.
3. Walking left to right, you accept each draft token as long as the target agrees with it. At the first disagreement you throw that token and everything after it away, and take the target’s own token for that position instead.

Even in the worst case, where the very first draft token is rejected, you still get one token out of the target pass, the same as normal decoding. In the best case, where all γ are accepted, the target’s prediction at the final position gives you a free extra token, so you get γ + 1 tokens from one pass of the big model.

The reason “entries” has to go, even if the target would have liked it, is that the draft chose it after “stale”. Once “stale” is replaced with “old”, every later guess was made about a sentence that no longer exists.

Why the output doesn’t change

If you’re decoding greedily, always taking the single most likely token, the argument is easy. Accept a draft token only if it’s the target’s top choice at that position. Every token you keep is one the target would have produced anyway, so the final text is identical, token for token.

Sampling is more interesting, and it’s the part I enjoyed. When you sample at a temperature, the target has a probability distribution p over the next token, and the draft has its own distribution q. The rule, from the two papers that introduced this in late 2022 and early 2023 (Leviathan, Kalman and Matias at Google, and Chen and colleagues at DeepMind, independently), is:

• Accept the draft’s token x with probability min(1, p(x) / q(x)).
• If you reject it, sample a replacement from the leftover distribution, max(0, p − q), renormalized.

So if the draft was less confident about a token than the target is, you always keep it. If the draft was overconfident, say q(x) = 0.6 where p(x) = 0.3, you keep it half the time. And when you reject, you draw from exactly the part of p that the draft was underweighting. Add the two cases up, the “accepted” mass and the “rejected and resampled” mass, and for every token you get precisely p(x). The output is a sample from the target model’s distribution, not an approximation of it. It’s the sort of rejection-sampling argument I last saw in a probability course during my math degree, and it’s nice to see it carrying real weight in production systems.

How much faster

It depends on how often the draft is right. Call the chance that a given draft token is accepted α. Then the expected number of tokens you get per target pass, with γ draft tokens, is

(1 − α^γ+1) / (1 − α)

With α = 0.8 and γ = 4, that’s (1 − 0.8⁵) / 0.2 = (1 − 0.328) / 0.2 ≈ 3.36 tokens per pass of the big model. But the draft isn’t free. If one draft step costs 5% of a target step, the round costs 1 + 4 × 0.05 = 1.2 target-passes’ worth of time, so the real speedup is about 3.36 / 1.2 ≈ 2.8x. Drop α to 0.6 and you get about 2.31 tokens per pass and 1.9x overall. That’s the whole game: a draft that agrees with the target more often, for as little cost as possible.

α varies a lot with what you’re generating. Code is predictable, especially boilerplate, closing brackets and the second half of a variable name. Prose that copies from the prompt is extremely predictable. A poem is not.

Where the draft comes from

The original version uses a separate small model, and that’s still the most common setup if you run models yourself. The catch is that you need a small model with the same tokenizer that thinks enough like the big one, and you’re now holding two models in memory.

The simplest version needs no model at all. Prompt lookup decoding, which got popular in 2023, drafts by searching the prompt for the last couple of tokens you generated and proposing whatever came after them last time. That sounds too dumb to work, and for chat it mostly doesn’t. For editing code it works very well, because “rewrite this function with one change” means the output is 95% a copy of the input, and the draft is right for long stretches at a time. Coding agents spend a lot of their time doing exactly that.

Then there are versions where the model drafts for itself. Medusa (early 2024) bolts several extra prediction heads onto the target, each guessing a token further ahead. EAGLE, from about the same time, trains a small head that drafts from the target’s own internal features instead of from the text, which gets much higher acceptance rates. DeepSeek-V3 was trained with a multi-token prediction objective at the end of 2024 and said the extra module could be reused for speculative decoding. That idea, training the drafter as part of the model, is now showing up in open model releases, and I’d expect it to be the normal thing within a year or so.

The hosted providers don’t usually say what they do, so I can’t tell you which of the frontier APIs use which version. Given how much of their cost is generation, I’d be surprised if any of them use none.

On my laptop

llama.cpp has supported a separate draft model in its server for a while. I tried it with a mid-sized open model as the target and the smallest model from the same family as the draft. Some rough numbers from my text file, with the caveat that it’s one machine and I wasn’t careful about anything:

On “refactor this C# method to use async”, about 1.7x faster. On “explain what this SQL does”, about 1.3x. On “write a short story about a lighthouse”, basically nothing, and once slightly slower, since the draft was wrong so often that its cost wasn’t paid back. Turning γ up from 4 to 8 helped the refactor and hurt the story.

That matches the theory better than my experiments usually do. The speedup lives in how predictable the output is, and for what I mostly ask models to do now, which is edit code, it’s predictable.

The thing I keep in mind when I read inference benchmarks is that speculative decoding speeds up a single stream, the case where hardware is otherwise waiting on memory. A server already batching hundreds of requests has its arithmetic busy, and spending some of it on guesses that get thrown away is a worse trade there. So “3x faster” on a laptop at batch size one and “3x faster” in a data center are two very different claims, and the press releases don’t always say which they mean.