Smaller numbers, same brain

In the intro to this series, we built a naive 26-billion-parameter model and calculated that it would need 400 GB of GPU memory. That's five datacenter GPUs just to hold it in memory.

Let's start shrinking it.

The first two optimizations are the oldest and most fundamental. One is about how the model remembers its conversation. The other is about how precisely it stores its numbers. Together, they cut our 400 GB in half - and one of them is a genuine "free lunch."

Taking Notes: The KV Cache

The first optimization we'll discuss doesn't save memory. It costs memory. But without it, transformer models would be so slow they'd be unusable.

To understand why, you need to know one thing about how transformers generate text: they produce one token at a time. Each time the model generates a new token, it needs to figure out which previous tokens are relevant to what it's about to say. It does this through something called "attention" - a mechanism where the new token looks back at every previous token and decides how much to care about each one.

Without any optimization, this means generating token number 1,000 requires the model to reprocess all 999 previous tokens through every layer (remember - a transformer is a stack of processing stages, each one refining the model's understanding) from scratch. Token 1,001 reprocesses all 1,000. Token 256,000 reprocesses all 255,999. Every single time.

The KV cache fixes this by taking notes. Here's what that looks like. Take the sentence "The wild horse jumped over the fence" and pretend each word is one token. For each token, the model computes a Key (a tag that helps future tokens find this one) and a Value (the information to deliver when found). They're not literal words - they're vectors of numbers - but intuitively:

Token	Key (findability tag)	Value (payload)
The	determiner, start of noun phrase	a specific thing follows
wild	adjective, modifying next noun	untamed, energetic quality
horse	noun, subject, animal	large animal, the actor
jumped	verb, past tense, physical motion	upward movement event
over	preposition, spatial	above-and-across trajectory
the	determiner, before object	a known thing follows
fence	noun, object, physical structure	barrier, what was cleared

That's one head's perspective - call it the grammar head. But there are 16 heads per layer, each reading the same sentence through a different lens. Here's the same sentence through two others:

A meaning head - less interested in grammar, more in what things are:

Token	Key (findability tag)	Value (payload)
The	function word, low information	(minimal)
wild	emotional intensity, nature-related	danger, freedom, unpredictability
horse	living creature, large mammal	strength, speed, four-legged
jumped	sudden movement, effort	explosive upward force
over	spatial relationship	clearance, success
the	function word, low information	(minimal)
fence	man-made boundary	confinement, obstacle

A narrative head - tracking who did what to whom:

Token	Key (findability tag)	Value (payload)
The	introduces main subject	first entity being set up
wild	character trait	the subject is untamed
horse	main character, agent	who the story is about
jumped	main event, climax action	what happened
over	event detail	direction of the action
the	introduces second entity	another thing coming
fence	obstacle	what was overcome

Same seven words, three completely different sets of notes. And those are all from the same layer - an early one, working close to the raw words. Here's what a grammar head in a late layer might produce for the same sentence, after the input has already been refined through dozens of earlier layers:

A grammar head in layer 28 (of 30):

Token	Key (findability tag)	Value (payload)
The	(absorbed into "horse" phrase)	(minimal - folded into noun)
wild	(absorbed into "horse" phrase)	(minimal - folded into noun)
horse	agent, completed successful action	powerful animal that overcame obstacle
jumped	completed event, physical feat	motion is finished, goal achieved
over	(absorbed into "jumped" phrase)	(minimal - folded into verb)
the	(absorbed into "fence" phrase)	(minimal - folded into noun)
fence	overcome obstacle	barrier that no longer matters

By layer 28, the model isn't thinking about individual words anymore. "The wild" has been absorbed into the concept of "horse." "Over the" has been folded into "jumped." The Keys and Values have become abstract - less about grammar and surface meaning, more about high-level roles in the narrative. This is why every layer needs its own KV cache: the same token means something different at every level of abstraction.

Three heads, two layers, and we've only scratched the surface. Multiply across all 16 heads and all 30 layers, and you start to see both why this is so powerful - and why it's so expensive.

When the model generates the next token, it compares that token's Query ("what am I looking for?") against every stored Key. High match? Pull in that token's Value. If the next word needs to know what moved, its Query matches the Key for "horse" (noun, subject, animal) and pulls in the Value (large animal, the actor).

And it's not one pair per token - it's one pair per token, per head, per layer. With 16 heads across 30 layers, that's 480 Key-Value pairs for every single token. Our seven-word sentence? 3,360 pairs. A 256,000-token conversation? 122 million pairs, costing 294 GB. That's where most of our 400 GB comes from.

The tradeoff is direct: speed for memory. Without the KV cache, the model would only need about 107 GB (just the weights and some working space) - but generating a response at the end of a long conversation would take an eternity, with latency scaling quadratically with context length. The cache stores all of these notes so they never need to be recomputed. Each new token computes its own Keys and Values, adds them to the cache, and looks up everything else from memory.

So the KV cache is the price of making inference practical. That 294 GB is the model's memory of the conversation.

Running total: still 400 GB. But now it actually runs at usable speed.

Using Smaller Numbers: FP16

We've been showing Keys and Values as intuitive labels - "noun, subject, animal" - but that's a cheat. In reality, each Key and Value is a vector: a list of numbers, a few hundred of them (256 per head in most of Gemma 4's layers). That "noun, subject, animal" Key for "horse" is really something like [0.82, -0.14, 0.03, 1.47, -0.91, ...] - 256 floating-point numbers that, together, encode the token's meaning in a way no human would recognize but the model can compute against at incredible speed.

Every one of those 256 numbers takes up memory. So does every one of the model's 26 billion parameters. Which raises a question: how precisely do you actually need to store each number?

In the early days of deep learning, these numbers were stored as 32-bit floating point values - FP32. Each number gets roughly 7 decimal places of precision, and each one takes 4 bytes of storage.

It turns out the difference between a weight of 0.0312547 and 0.03125 doesn't meaningfully affect the model's output. FP16 - half precision - stores each number with about 3-4 decimal places in 2 bytes. Half the precision, half the storage.

This might sound like a crude hack. When a layer multiplies millions of inputs by millions of weights and sums them up, small rounding errors in individual weights cancel out statistically. The overall behavior of the layer is preserved even when individual values are slightly less precise.

FP16 halves everything - both the model weights and the KV cache:

Component	FP32	FP16
Model weights	104 GB	52 GB
KV cache (256k context)	294 GB	147 GB
Overhead	2.6 GB	2.6 GB
Total	~400 GB	~202 GB

Cut in half. Barely any quality loss. That's the free lunch.

Sidebar: FP16 wasn't obvious

FP16 for inference was a discovery, not an assumption. Researchers had to prove that lower precision wouldn't cause models to fall apart. NVIDIA had to ship dedicated hardware support for it - Tensor Cores on the V100 GPU in 2017. Mixed-precision training (FP16 forward pass, FP32 for gradient accumulation) was a research paper (Micikevicius et al., 2018) that had to demonstrate it actually worked. None of this was a given beforehand.

The Precision Journey

Here's something worth appreciating: modern models don't even train in pure FP32 anymore. They use "mixed precision" - FP16 for the heavy computation, FP32 only for the sensitive parts.

During each training step, the model:

1. Casts weights to FP16 for the forward pass - the step where input flows through the model to produce output (fast)
2. Computes gradients - the tiny adjustments calculated at each training step - in FP16
3. Accumulates those gradients back into an FP32 "master copy" of the weights

If that sounds like an accountant who does quick mental math all day but keeps a precise ledger on their desk - that's basically what it is.

That FP32 master copy exists because training adjusts weights by impossibly tiny amounts - a change in the 7th decimal place. If you accumulate thousands of these tiny adjustments in FP16, the rounding errors compound and the total drifts. But the bulk of the expensive computation happens in FP16, which is 2-8x faster on modern GPUs thanks to those dedicated Tensor Cores.

When training is done, the FP32 master copy gets thrown away. The FP16 weights are what gets shipped. And those are what eventually get compressed even further for your GPU - but that's a later post.

Sidebar: The quiet format war

The move to lower precision sparked a format skirmish. Google designed bfloat16 for its TPU chips - a 16-bit format that keeps FP32's range but sacrifices precision. Google's logic: neural networks care more about range than precision. NVIDIA's V100 had bet on standard IEEE FP16 with its Tensor Cores. Three years later, NVIDIA's A100 quietly added bfloat16 support - a tacit acknowledgment that Google had the right idea about the tradeoff.

The journey of a single weight's precision:

FP32 (training)  -->  FP16 (shipped model)  -->  Q4 (4-bit, on your GPU)

We're at step two. The model is 202 GB. Still way too big for consumer hardware, but we just eliminated roughly 200 GB with a conceptually simple idea: use smaller numbers.

Next up, we tackle the other half of the problem. That 147 GB of KV cache memory is about to get demolished.

Step	Weights	KV Cache	Overhead	Total
Naive (FP32)	104 GB	294 GB	2.6 GB	~400 GB
+ FP16	52 GB	147 GB	2.6 GB	~202 GB

---

Next up: Sharing the Textbook: How GQA Crushed the KV Cache | Series start: How 400 GB Became 18

Written by Claude as dictated by Harald. Claude's own numerical precision is, unfortunately, classified.