The finishing touches

We're at 19 GB. Down from 400 GB. The major optimizations are done - FP16, GQA, sliding window, MoE, and quantization got us from "needs a datacenter" to "fits on a consumer GPU."

Gemma 4 adds two more innovations of its own. Neither is as dramatic as what came before, but they complete the picture of how Google squeezed maximum capability out of minimum hardware.

K=V Equivalence: One Folder Instead of Two

In a standard transformer, each attention head stores two things per token: a Key ("what do I contain?") and a Value ("what information do I carry?"). These are separate vectors, independently computed and separately stored.

Remember the horse? After GQA, we had one shared Key and Value across all heads:

Token	Shared Key	Shared Value
horse	primary actor	powerful animal in motion

The Key helps future tokens find the horse. The Value is what they get when they find it. Two separate vectors, two separate roles.

Gemma 4 discovered that for global attention layers, these can be identical. How you'd describe the horse to find it is already the useful information about it. Set K equal to V, store one copy, use it for both purposes:

Token	K = V
horse	powerful animal driving the action

One vector. It answers "what do I contain?" and "what should I deliver?" with the same representation - because for global attention layers, those were always the same question.*

*As always, these aren't literal text labels - they're vectors of hundreds of numbers that encode something more like a vibe than a sentence. "Powerful animal driving the action" is our best attempt to put words on what [0.82, -0.14, 0.03, 1.47, ...] means.

How Big Is the Savings?

K=V only applies to global attention layers - but those are exactly the ones that matter. Remember, after sliding window attention, 25 of the 30 layers cache only a small fixed window of tokens. Their memory is constant regardless of context length. The 5 global layers are the ones that cache the entire 256k context - they're the bottleneck that determines how long a conversation the model can hold.

Halving them cuts the KV cache from ~1.4 GB to ~0.7 GB. About 700 MB in absolute terms. Not dramatic.

But there's another way to look at this. We've spent the whole series establishing that KV cache is context memory - the model's ability to remember the conversation. Halving the KV cache for global layers doesn't just save 700 MB. It means you could double the context window for the same memory budget. Or put differently: on a memory-constrained GPU where context length is the bottleneck, K=V turns a 256k context into a 512k context for free.

700 MB of savings sounds modest. Doubling your context window sounds like a feature.

K=V is valuable because it's free. Zero quality loss, no inference overhead, just a mathematical equivalence that Google proved holds. In a world where every megabyte matters on consumer hardware, free optimizations are always welcome.

Sidebar: Career advice

If you've been reading this series thinking "these optimizations don't sound that clever" - good news, you might have a future in AI research. The field is hiring.

In 2025, Meta CEO Mark Zuckerberg personally emailed researchers offering $10 million per year. Google DeepMind countered with $20 million packages. One researcher reportedly declined an $18 million annual offer.

I'm a game developer by trade, so when Claude told me that one of the key innovations was "just use 16-bit floats instead of 32-bit," I thought it was pulling my leg. No self-respecting game dev would even consider using an FP32 where an FP16 will do - and we'd still prefer a fixed-point integer if we can get away with it. Meanwhile, one of the innovations in this post amounts to "set K equal to V." I'm in the wrong line of business. Not that I'm bitter.

Per-Layer Embeddings: Flash Storage for Tiny Models

The second Gemma 4 innovation is Per-Layer Embeddings (PLE), and it targets a completely different problem. PLE is exclusive to the edge models - the E2B and E4B variants designed for phones, tablets, and browsers.

The Problem with One Embedding

In a standard transformer, every token starts the same way. The word "bank" gets converted to a single embedding vector - a list of numbers representing its meaning - and that same vector enters every layer of the model. Layer 1 (basic syntax) and layer 28 (contextual meaning) both start from the same representation and have to do all the work of specializing it themselves.

Large models have enough capacity to handle this. With billions of parameters per layer, they can transform a generic starting point into whatever specialized representation each layer needs. Small models don't have that luxury. With fewer parameters, each layer has less room to work with, and starting from a one-size-fits-all embedding wastes some of that precious capacity on transformations that could have been baked in from the start.

Per-Layer Hints

PLE's fix: give each layer its own embedding table. Instead of one table that maps each token to one vector, PLE maintains a separate table for every layer. Each layer gets a hint tailored to what that layer cares about.

Back to the horse. Remember how different layers see different things? Layer 1 cares about grammar, layer 28 cares about narrative roles. Without PLE, both layers receive the same generic embedding for "horse" and have to spend capacity figuring out what to focus on. With PLE:

Layer	Hint for "horse"
Layer 1 (syntax)	noun, singular, animate
Layer 15 (semantics)	large mammal, domesticated, four-legged
Layer 28 (narrative)	agent, protagonist, overcame obstacle

These hints aren't hand-designed - they're learned during training, just like every other parameter in the model. The training process figures out what each layer needs to hear about each token and bakes it into the embedding tables. At inference time, layer 1 doesn't waste parameters rediscovering that "horse" is a noun. Layer 28 doesn't have to rebuild the concept of "agent" from a generic starting point. The hint gets each layer closer to where it needs to be.

In a large model, this isn't worth the trouble. A 26B model has enough capacity per layer to derive those specializations on its own, so the quality gain from hints is negligible. And large models run on GPUs with plenty of VRAM, so the flash storage trick - PLE's other superpower - solves a problem that doesn't exist. You'd be adding a gigabyte of tables for almost no benefit.

But in a 2B model running on a phone, both sides of the tradeoff flip. The layers are too small to waste capacity on rediscovery, so the hints genuinely help quality. And VRAM is so scarce that storing those hints on flash instead of in memory is the difference between fitting and not fitting. PLE is an optimization that only makes sense at the edge - which is exactly where Google deployed it.

The Flash Storage Trick

But all those separate embedding tables are large. The vocabulary alone is 262,000 tokens, and each table maps every token to a 256-dimensional vector. At full precision, the PLE tables add up to about 4.5 GB - but quantized to 4-bit (like the rest of the model on-device), they're closer to a gigabyte on disk.

But look at the access pattern. When the model generates text, it produces one token at a time - each word depends on the previous ones, so they come out sequentially. And you only need the hint for the current token. Previous tokens have already been processed and their results are sitting in the KV cache - the hints did their job and can be forgotten. So for each new token, you need exactly one row from each layer's table - a few KB per token. A tiny read.

A few KB is nothing for a modern SSD. Even a phone's flash storage can serve that in microseconds - well within the time the GPU spends computing the layer. So PLE stores the embedding tables on flash storage instead of in VRAM. They get fetched on demand, one row at a time, and discarded after use.

The full tables never touch GPU memory. A gigabyte of quantized parameters that exists on disk, delivers its value through pinhole-sized reads, and costs essentially zero VRAM.

"Effective" Parameters

This is why the E2B and E4B use "E" for "effective" parameters. The E2B has 2.3 billion effective parameters - what actually lives in VRAM - but 5.1 billion total parameters on disk. The E4B has 4.5 billion effective, roughly 8 billion total. The difference is entirely PLE tables.

The naming is honest: these models have the VRAM footprint of a 2B or 4B model, but the representational depth of something much larger. The E2B fits in about 2 GB of VRAM (at 4-bit quantization) while achieving quality that rivals traditional models several times its effective size.

Why This Matters

PLE doesn't apply to the 26B-A4B or 31B models, so it doesn't change our running total. But it's a striking example of the principle that drives this whole series: not all parameters need to be in GPU memory at all times.

PLE moves rarely-used data to slower storage. MoE keeps inactive experts loaded but not computing. Sliding window bounds what each layer needs to remember. The KV cache itself is a tradeoff between memory and recomputation. Different strategies, same insight: the memory hierarchy has multiple tiers, and the art is putting each piece of data on the cheapest tier it can tolerate.

The Final Tally

One last time - here's the full journey, naive to actual:

Step	Year	Weights	KV Cache (256k)	Overhead	Total
Naive (FP32, full MHA)	pre-2018	104 GB	294 GB	2.6 GB	~400 GB
+ FP16	~2018	52 GB	147 GB	2.6 GB	~202 GB
+ GQA	2023	52 GB	~58 GB	2.6 GB	~112 GB
+ Sliding window	2023	52 GB	~5.5 GB	2.6 GB	~60 GB
+ MoE	2023-24	52 GB	~5.5 GB	2.6 GB	~60 GB (6.5x faster)
+ Q4_K_M + Q4 KV	2022-23	14.8 GB	~1.4 GB	2.6 GB	~19 GB
+ K=V equivalence	2026	14.8 GB	~0.7 GB	2.6 GB	~18 GB

400 GB to 18 GB. A 22x reduction. Eight years of compounding improvements, from "needs a datacenter" to "runs on your desk."

But these optimizations only explain how Gemma 4 fits in memory. They don't explain why a 26B model outscores GPT-4o on the Arena leaderboard - a model backed by one of the richest companies on earth with undisclosed but presumably enormous parameter counts. That's a story about training: better data curation, distillation (training a small model to mimic a larger teacher), and RLHF (fine-tuning based on human ratings). The architecture makes it possible to run. The training makes it good. Parameters set the ceiling; everything else determines how close you get to it.

And the story isn't over. In the next and final post, we'll look at research that might make GPU memory irrelevant entirely.

---

Next up: Models Bigger Than Memory: The Future of Consumer AI | Series start: How 400 GB Became 18

Written by Claude as dictated by Harald.