Mixture of experts, visually

46.7 billion parameters, and 12.9 billion of them used per token. Those were Mistral’s numbers for Mixtral 8x7B when it came out at the end of 2023, and I remember reading them and thinking one of them had to be a typo, because “8x7B” suggests 56 and neither number is 56. They’re both right, and working out why is a good way into the idea behind most of the large open-weights models of the last two years: DeepSeek-V3 and R1, Llama 4, Qwen3’s big models, OpenAI’s gpt-oss. They’re all mixture-of-experts models. I wrote about the DeepSeek release last January and waved my hands at the “671 billion total, 37 billion active” part, so this is me paying that off.

Where the parameters are

In the transformer post I drew a transformer block as two halves. Attention is where tokens look at each other. Then each token’s vector goes, on its own, through a feed-forward network (FFN): a couple of big matrix multiplies with a nonlinearity in between, which takes the 4,096 numbers that represent a token, expands them to something like 14,000, and squeezes them back down. Nobody fully understands what the FFN does, but the usual description is that a lot of what the model “knows” is stored there, and it’s where most of the parameters are.

For Mixtral, which uses the gated FFN that Llama made standard, one FFN has three matrices of 4,096 × 14,336. That’s

3 × 4,096 × 14,336 = 176,160,768

parameters, about 176 million, per FFN. Attention in the same block is about 42 million, a quarter as much. So in a dense model of this shape, four-fifths of the work per token is the FFN.

A mixture-of-experts layer replaces that one FFN with several of them, the experts, plus a tiny network called the router that picks, for each token, which experts it goes through. Mixtral has 8 experts per layer and sends each token through 2.

Now the two Mixtral numbers add up. Thirty-two layers, each with 8 experts of 176 million parameters, is about 45.1 billion in experts. Add 1.34 billion for attention across all the layers and about 0.26 billion for the embedding tables at either end, and you get 46.7 billion. “8x7B” double-counts the attention and embeddings, which are shared, which is why it isn’t 56. For one token, only 2 experts per layer run: 2 × 176 million × 32 is 11.3 billion, plus the same 1.6 billion of shared parts, and that’s 12.9 billion.

The router is almost nothing by comparison. It’s one small matrix per layer, 4,096 × 8, about a million parameters across the whole model.

How the router decides

It’s a linear layer and a softmax, which is about as simple as a learned decision gets. The token’s vector is multiplied by the router’s matrix to get one score per expert. Mixtral keeps the top two scores, throws away the rest, and runs a softmax over the two it kept to turn them into weights that add up to one. Say a token’s eight scores come out as

0.3, 2.1, −0.5, 0.9, 0.0, 1.7, −1.2, 0.4

The top two are experts 2 and 6, as in the diagram. The softmax over just those two is e^2.1 / (e^2.1 + e^1.7) = 8.17 / 13.64 ≈ 0.60 for the first, and 0.40 for the second. The token goes through both experts, and the layer’s output is 0.60 times the first expert’s output plus 0.40 times the second’s. The other six never see the token at all, which is the whole point: their weights aren’t touched, so they cost nothing for this token.

Different models make different choices here. Google’s Switch Transformer paper in 2021 went all the way to one expert per token. gpt-oss-120b has 128 experts per layer and picks 4. DeepSeek-V3 has 256 small experts per layer and picks 8, plus one “shared” expert that every token goes through regardless, the idea being that the shared one picks up the common stuff and the routed ones can be more specialized. The trend since Mixtral has been towards more, smaller experts, and the arithmetic says why: picking 8 of 256 gives you vastly more possible combinations than 2 of 8, at the same cost per token.

Every token makes this choice separately, at every layer. A token can go through experts 2 and 6 in layer 1 and experts 3 and 8 in layer 2, and the next token in the same sentence can take a completely different path.

Why it’s cheaper

The compute for a forward pass is, to a good approximation, two floating-point operations per parameter used, one multiply and one add. So a Mixtral token costs about 2 × 12.9 billion ≈ 26 billion operations, against about 93 billion for a dense model with all 46.7 billion parameters. You get the capacity of the big model, or something close to it, at roughly the compute of a 13B one.

It also helps during training, which is really where this came from. Noam Shazeer and colleagues at Google had a 2017 paper with the very 2017 title “Outrageously Large Neural Networks”, which put thousands of experts between LSTM layers, and Google’s GShard and Switch Transformer work in 2020 and 2021 brought the idea to transformers. The argument then was the same as now: for a fixed budget of compute, a sparse model with many more parameters learns more than a dense one.

Here’s what the numbers look like for a few of the models I mentioned.

What it doesn’t save you

Memory. Every expert has to be loaded somewhere, because the next token might want any of them. Mixtral at 16 bits per parameter is about 93 GB of weights. gpt-oss-120b is shipped with its expert weights already quantized to about four bits, which is the only reason it fits on a single 80 GB GPU. DeepSeek-V3 is a multi-GPU model no matter what you do.

This is where MoE gets interesting for running models yourself, and a bit counterintuitive. When you generate text one token at a time on a single machine, the slow part usually isn’t the arithmetic. It’s reading the weights out of memory for every token. A dense 47B model reads all 47 billion parameters for each token it writes. Mixtral reads only the 12.9 billion it’s using. So an MoE model that fits in memory generates text at something like the speed of a model its active size, and that’s why these models are so popular with people running them on machines with a lot of unified memory. You need the RAM for the big model and you get the speed of the small one.

Serving at scale is a different story, and it took me a while to see it. A provider batches many users’ tokens together, and with enough tokens in a batch, every expert in every layer is wanted by somebody, so all the weights get read anyway. What’s saved there is compute, not memory traffic. And the experts often don’t fit on one GPU, so they’re spread across several, which means tokens have to be shipped to whichever GPU holds their expert and the results shipped back, in every layer. That all-to-all shuffle is one of the main engineering problems in serving these models, and it’s the kind of thing that only matters when you’re running thousands of GPUs, which I’m not.

Keeping the experts busy

Left alone, training goes wrong in a predictable way. Early on, a couple of experts happen to be slightly better. The router learns to send more tokens to them, so they get more training and get better, so the router sends them even more, and before long a few experts do everything and the rest are dead weight. It’s a rich-get-richer loop, and you can see it coming from the structure of the thing.

The standard fix is an extra term in the loss that punishes imbalance. Switch Transformer’s version is neat enough to write down. For each expert i, let f_i be the fraction of tokens in the batch that were actually sent to it, and P_i be the average probability the router gave it. The extra loss is

α · N · Σ f_i · P_i

where N is the number of experts and α is a small weight. If everything is perfectly balanced, every f_i and P_i is 1/N, the sum is N × 1/N², and the whole thing comes out to exactly α. Any imbalance makes it bigger. (I like that it’s normalized so the best case is a clean constant, though α is still a hand-tuned knob like every other one.) DeepSeek-V3 went a different way and mostly did without the auxiliary loss (they kept a very small one as a backstop), nudging a per-expert bias on the router scores up or down depending on whether that expert is over- or under-used.

There’s also a capacity limit in a lot of implementations. Each expert gets a fixed number of slots per batch, roughly the batch size divided by the number of experts, times a fudge factor a bit over one. If more tokens than that want the same expert, the extras are dropped from that layer and just carry on through the residual connection as if the layer weren’t there. That sounds alarming and in practice works out fine, which is a sentence I seem to write a lot about neural networks.

The experts aren’t experts

This is the part that surprised me. I’d assumed, from the name, that there’d be a math expert and a French expert and a Python expert, and that the router was a kind of dispatcher sending questions to the right department. When the Mixtral paper looked at which experts tokens went to, it didn’t find that. On text from very different sources, ArXiv papers, biology, philosophy, math, the routing looked broadly similar. What it found was more mechanical: the same token tends to go to the same experts, consecutive tokens often go to the same ones, and in code, indentation tokens and keywords like self cluster.

So the specialization is real, but it’s at the level of syntax and token types, not subjects, and that matches what I’ve read elsewhere. “Expert” is a word from the 1990s mixture-of-experts literature, which really did mean separate specialist models with a gate in front, and it stuck. If I were naming it now I’d call them shards, or lanes, or anything less flattering. People hear “experts” and imagine a committee, and then draw conclusions about these models that the routing data doesn’t support.

Where it leaves me

Nearly every open-weights model I’ve tried this year has been an MoE, and as far as I can tell from what’s been published, that’s where most of the large models are going. Whether the closed frontier models are built this way isn’t something their makers say, though people seem to assume it.

The practical thing I take away is to read a model card with both numbers in mind. Total parameters tell you how much memory you need. Active parameters tell you, roughly, how fast it’ll run on one machine and how much it costs per token to serve. A “120B” model with 5 billion active is a very different thing to run from a dense 120B.