The hospital with 128 doctors

We're at 60 GB. Three optimizations - FP16, GQA, and sliding window attention - crushed the KV cache from 294 GB to under 6 GB. The problem has flipped: nearly all of the remaining memory is model weights. 52 GB of parameters sitting in GPU memory, and every single one of them activates for every single token.

Mixture of Experts changes that equation. But not in the way you might expect.

The Naive Hospital

Our current model is like a hospital with one doctor who knows everything. Every patient - regardless of whether they have a broken arm or a headache - goes through the same doctor, the same examination process, the same everything. That doctor is very good (26 billion parameters worth of good), but every patient takes the full doctor's time.

What if, instead, you had 128 specialist doctors and a triage nurse?

How MoE Works

In a Mixture of Experts model, the dense feed-forward layers (the parts of each transformer layer that do the heavy number-crunching) are replaced by a collection of smaller "expert" sub-networks. A lightweight router - the triage nurse - looks at each incoming token and decides which experts to send it to.

Gemma 4's 26B-A4B variant has:

• 128 expert sub-networks
• 1 shared expert that's always active
• A router that picks 8 experts per token

So for each token, only 8 out of 128 experts fire, plus the shared one. About 4 billion parameters do actual computation out of 26 billion total.

Our horse sentence makes this visible. The router looks at each token and assembles a different team:

Token	Some of the 8 experts activated
The	syntax, phrase structure, article usage, ...
wild	adjectives, emotion, nature/outdoor, ...
horse	animals, physical entities, agent tracking, ...
jumped	motion verbs, past tense, physical actions, ...
over	spatial relations, prepositions, trajectory, ...
the	syntax, phrase structure, article usage, ...
fence	physical objects, structures, obstacles, ...

"Horse" doesn't need the code expert or the math expert. It needs the animal expert, the entity tracker, the physical-world expert. "Jumped" needs a completely different team. The router figures this out automatically - it learned during training which experts are good at what. Notice that "The" and "the" route to roughly the same team: same word, same needs.

The "A4B" in the model name? It stands for "Active 4 Billion" - the number of parameters that actually do work per token.

The Catch

MoE doesn't save memory.

All 128 doctors have to be in the building. Even if only 8 see any given patient, their offices are still there, their equipment is still there, they're still on payroll. All 26 billion parameters must be loaded into GPU memory. You just don't compute through most of them for each token.

Before MoE: 52 GB of weights, all 26B active per token
After MoE:  52 GB of weights, only 4B active per token

Same memory. But the expert layers - the big feed-forward blocks that dominate compute - do about 6.5x less work per token. The actual end-to-end speedup is less than 6.5x, because attention layers, the shared expert, the router, and layer norms still run for every token. But the expert layers are where most of the compute lives, so the real-world speedup is still dramatic.

So why include MoE in a series about fitting a model on consumer hardware? Because of what it makes possible.

The Offloading Revolution

My GPU has 12 GB of VRAM. The model needs 18 GB. That means about a third of the weights have to spill into regular system RAM, connected to the GPU via PCIe (the physical connection between your CPU and GPU) - a much slower bus than the GPU's internal memory bandwidth.

With a dense model, this is devastating. Every token activates every parameter. If some layers are on CPU, you're shoveling data across the PCIe bus for every single token you generate. That's the 2-5 tokens-per-second territory. Unusable for interactive work.

With MoE, most of those offloaded experts just sit idle for any given token. If 6 of the 8 selected experts happen to be in VRAM and 2 are in system RAM, you only pay the PCIe transfer cost for those 2. And the next token might route to a completely different set of experts - maybe all 8 are on the GPU this time.

Dense offloading: every token calls every doctor, some doctors are in a building across town, every patient waits for the commute.

MoE offloading: each token only needs 8 doctors, most are in this building, occasionally one is across town.

That's why Gemma 4 26B-A4B benchmarks at ~44 tokens per second on a 12 GB consumer GPU with partial offloading. A dense 26B model in the same configuration? You'd be lucky to get 5.

MoE didn't shrink the model. It made the model survivable when it doesn't fully fit.

Better Quality, Not Just Better Speed

There's another angle. The 26B-A4B achieves roughly 97% of the quality of Gemma 4's dense 31B model on standard benchmarks. A dense 4B model would be significantly worse. So where does the extra quality come from if only 4B parameters are active?

From specialization. Different experts learn different things. One expert might get really good at code. Another at conversational language. Another at mathematical reasoning. The router learns which experts to invoke for each kind of token, so the model dynamically assembles the right team of specialists for each situation.

A dense 4B model is like one doctor who's decent at everything. An MoE with 4B active parameters is like 8 elite specialists collaborating on each case, drawn from a pool of 128. Same compute budget, vastly different capability.

Sidebar: The Chinchilla paper

In 2022, a team at DeepMind published a paper called "Training Compute-Optimal Large Language Models" - better known as the Chinchilla paper. Its thesis was blunt: the entire industry was building models wrong.

The culprit was OpenAI's influential scaling laws paper (Kaplan et al., 2020), which concluded that model performance was most sensitive to parameter count. The takeaway: for a fixed compute budget, make the model bigger rather than train it longer. Everyone followed that playbook - GPT-3 was 175 billion parameters trained on just 300 billion tokens.

Chinchilla said the ratio was off. The optimal balance was roughly 20 training tokens per parameter. By that rule, GPT-3 should have either been trained on 3.5 trillion tokens, or been much smaller for its dataset. Most models were massively undertrained relative to their size.

The proof was a 70-billion-parameter model (Chinchilla) trained on 1.4 trillion tokens. It outperformed Gopher, DeepMind's own 280-billion-parameter model trained on 300 billion tokens. Four times smaller, four times more data, better results. The implication: the industry had been spending billions of dollars on parameters it didn't need.

One of the co-authors was Arthur Mensch. In 2023, he left DeepMind, co-founded Mistral AI with two ex-Meta researchers who had helped build LLaMA, and in December 2023 released Mixtral 8x7B - the first major open-source MoE model. It was competitive with GPT-3.5 on major benchmarks while using a fraction of the active parameters. The man who proved everyone was building models too big went on to build a smaller, smarter one and gave it away for free.

Since then, the industry has pushed even further than Chinchilla suggested - modern small models train on far more than 20 tokens per parameter, deliberately "overtraining" to squeeze maximum quality out of fewer weights. Chinchilla said "train longer." The industry said "yes, and then keep going."

Where We Stand

Step	Weights	KV Cache	Overhead	Total
Naive (FP32)	104 GB	294 GB	2.6 GB	~400 GB
+ FP16	52 GB	147 GB	2.6 GB	~202 GB
+ GQA	52 GB	~58 GB	2.6 GB	~112 GB
+ Sliding window	52 GB	~5.5 GB	2.6 GB	~60 GB
+ MoE	52 GB	~5.5 GB	2.6 GB	~60 GB (6.5x faster)

The total hasn't changed. But the expert layers do a fraction of the work per token, and - critically for consumer hardware - the model can survive partial offloading to system RAM without becoming unusable.

We have one more class of optimization to apply, and it's the one that finally attacks those 52 GB of weights directly. It's also the one that sounds the most absurd: what if we stored each of the model's 26 billion numbers using only 16 possible values?

---

Next up: Sixteen Shades of Weight: Quantization and PolarQuant | Series start: How 400 GB Became 18

Written by Claude as dictated by Harald. Claude declined to disclose how many of its experts were active during writing.