Sixteen shades of weight

We're at 60 GB. The KV cache is solved. MoE makes inference fast enough for real-time use and offloading survivable. But 52 GB of model weights are still sitting there, stored in FP16 - 65,536 possible values per number.

Quantization asks: how many of those values do you actually need?

The Training-Inference Gap

To understand why aggressive compression works for inference but not training, you need to appreciate what's happening in each phase.

During training, the model adjusts its weights by impossibly small amounts called gradients. A single training step might nudge a weight from 0.031254782 to 0.031254919 - a change in the 7th decimal place. If you're storing that weight as 0.03, a tweak in the 7th decimal place won't budge it. The gradient disappears. The model can't learn because every adjustment gets rounded away.

That's why training needs precision. FP16 (65,536 values) is the practical minimum, with FP32 (4.3 billion values) kept as a master copy for gradient accumulation.

But once training is done, the weights are frozen. Nobody's making tiny adjustments anymore. The model just multiplies inputs by weights and adds them up, billions of times per token. Individual weight precision doesn't matter much. What matters is the statistical behavior across millions of weights.

If quantization rounds some weights up and some down, the errors cancel out across millions of multiplications. The final output barely changes because the noise is random and symmetric. It's the law of large numbers working in your favor.

It's like drawing a smooth gradient. While you're painting, it's great to have millions of colors to blend with. But once you're done, you can reduce to just 16 colors and use dithering - scattering slightly-wrong pixels so the eye sees the right average. Each pixel is off, but the overall impression is spot on. Quantization is dithering for neural networks.

What Quantization Looks Like

Quantization applies to any block of numbers in the model - weights, Keys, Values, anything stored as a list of floats. To see what it does concretely, take the 256-number Key vector for "horse" from the KV cache post. In FP16, it looks something like:

[0.82, -0.14, 0.03, 1.47, -0.91, 0.28, -0.55, 0.09, ...]  (256 numbers)

At Q4 - 16 possible values - each number gets rounded to the nearest bucket:

[0.80, -0.13, 0.00, 1.47, -0.93, 0.27, -0.53, 0.13, ...]  (256 numbers)

Every single number is slightly wrong. But some rounded up, some rounded down, and across 256 dimensions the errors largely cancel out. When a Query asks "what's the grammatical subject?", it gets essentially the same answer from both vectors - because the direction the vector points in barely changed, even though every individual coordinate shifted a little.

The sweet spot turns out to be Q4 - 16 possible values per weight, about 0.57 bytes each. That gets our 52 GB of weights down to 14.8 GB, and it's what the gemma4:26b download on Ollama uses by default. Go lower than 16 values and the errors stop cancelling out - the model starts producing mangled grammar and confident nonsense.

And remember our old friend the KV cache? The one that started at 294 GB and we've been beating up for the entire series? Quantization applies there too. Those Key and Value vectors for every token - the "primary actor" and "powerful animal in motion" - get the same treatment. 5.5 GB down to 1.4 GB. At this point we're almost bullying it.

Standard Quantization: Good Enough, Not Optimal

Standard quantization methods like Q4_K_M use roughly uniform spacing across the range of weight values. Take the minimum and maximum weight in a block, divide the range into 16 evenly-spaced buckets, round each weight to the nearest bucket.

This works, but it's wasteful. Neural network weights aren't evenly distributed. They cluster in a bell curve (Gaussian distribution) around zero. Uniform buckets waste resolution on the tails where almost no weights live, and cram too many weights into too few buckets near the center where most of the action is.

Back to our gradient: standard quantization is like dividing each RGB channel into 6 evenly-spaced levels. Simple, but dumb - it wastes palette slots on colors that barely appear in the image.

PolarQuant: Smarter Buckets

PolarQuant fixes this with a clever trick. There are two PolarQuant papers (AISTATS 2026 for KV cache, 2026 for weights), both using the same core idea: don't quantize the weights as-is. Transform them into a known distribution first, then quantize optimally for that distribution.

Remember our gradient? Here's the same 16 colors, but picked from the blues and golds that actually appear in the image instead of spread evenly across all of RGB:

Same number of colors. The smart palette just doesn't waste any on reds and greens that aren't in the picture.

Editor's note: while generating this image, I asked Claude multiple times if that bottom gradient was really 16 colors. It assured me it was. I asked it to check. It checked and said yes. I asked it to check again. It checked again and said yes. Could I open it in MS Paint and count the colors myself? Sure. But where's the fun in that? Where's the mystery? Where's the adventure? I choose to trust the machine.

For the weights, PolarQuant does two things: a mathematical rotation (a Hadamard transform) that reshapes the weight distribution, then optimal bucket placement for the resulting shape. To understand why this works - and why one of these steps matters far more than the other - we need to look at what actually goes wrong during standard quantization.

The Real Villain: Outliers

Neural network weights mostly cluster near zero, but a few are much larger - maybe 0.1% of weights have values 10x or 20x the average. These outliers stretch the quantization range. If most weights are between -1 and 1, but a handful reach 8 or -6, your 16 uniform buckets have to cover -8 to 8. Each bucket spans a width of 1.0. Most weights - the 99.9% living between -1 and 1 - are crammed into just 2-3 buckets. Thirteen buckets sit nearly empty, covering tail space where almost nothing lives.

The outliers hijack the entire quantization scheme. Not by being wrong themselves, but by ruining precision for everything else.

The Rotation: Spreading the Load

PolarQuant's key insight is that you can fix this before quantization ever happens. The Hadamard transform is a mathematical rotation applied to the weight matrix. Here's what that means concretely. Say one dimension of a weight block has an outlier:

[8.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]

The rotation redistributes this value across all dimensions:

[0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9]

The information from the outlier isn't lost - it's spread across many coordinates instead of concentrated in one. After rotation, the whole distribution becomes a clean bell curve with no extreme tails, no range-stretching outliers. Once you know it's a bell curve, you can place your quantization buckets where the weights actually are - more near zero where they're dense, fewer in the sparse tails.

Why This Doesn't Break the Model

This probably sounds alarming. We just changed all the weight values. How can the model still work?

Because a rotation is a change of coordinate system, not a change of information. Think of a house that's 100 meters north and 1 meter east of you. In north/east coordinates, that's (100, 1). Rotate your coordinate system 45 degrees and the same house is at roughly (71, 70). The house didn't move. You're just describing its position differently.

The math guarantees this: if you rotate the weights, you can apply the inverse rotation to the inputs, and the output is mathematically identical. Not approximately - exactly. Before quantization, the rotation changes absolutely nothing about what the model computes.

This isn't a mathematically precise depiction of a Hadamard rotation - just a visual intuition for how the same data can look very different from a different coordinate system.

Once the outliers are gone, the quantization range shrinks dramatically. Your 16 buckets cover -2 to 2 instead of -8 to 8. Every weight gets 4x more precision automatically - even with dumb uniform spacing.

And that's why the rotation alone accounts for 98% of the quality improvement, while optimal bucket placement contributes only 2%. The problem with standard quantization was never "buckets in the wrong places." It was "outliers distort the entire scheme." Fix the outliers by spreading them out, and even simple quantization works almost as well as the mathematically optimal version. The smart bucket placement is just the last bit of polish.

The Results

PolarQuant Q5 (32 values per weight) achieves a perplexity of 6.39, versus 6.37 for full FP16. Perplexity measures how well a model predicts text - lower is better, and a difference of 0.02 is negligible. That's near-lossless - better than standard Q5_K_M at the same file size.

The technique extends to the KV cache too. Applying the same rotation-then-quantize approach to the KV cache achieves Q3 quantization - just 8 values per entry - with no accuracy loss. Standard Q3 on the KV cache would degrade quality. PolarQuant's smart bucketing makes it viable.

There's even a PolarQuant Q5 variant of the full Gemma 4 31B dense model on HuggingFace - which might make the 31B dense model viable on consumer hardware. We haven't tried it yet, but the math says it should be close.

The Final Push

You know the drill by now. Applying Q4_K_M to weights and Q4 to the KV cache:

Step	Weights	KV Cache	Overhead	Total
Naive (FP32)	104 GB	294 GB	2.6 GB	~400 GB
+ FP16	52 GB	147 GB	2.6 GB	~202 GB
+ GQA	52 GB	~58 GB	2.6 GB	~112 GB
+ Sliding window	52 GB	~5.5 GB	2.6 GB	~60 GB
+ MoE	52 GB	~5.5 GB	2.6 GB	~60 GB (6.5x faster)
+ Q4_K_M weights	14.8 GB	~5.5 GB	2.6 GB	~23 GB
+ Q4 KV cache	14.8 GB	~1.4 GB	2.6 GB	~19 GB

From 400 GB to 19 GB. Twenty-one times smaller.

That 19 GB is close to what the gemma4:26b-a4b-it-q4_K_M model is on Ollama, listed at 18 GB. The remaining gap comes from the Gemma 4-specific optimizations we'll cover in the next post.

But take a moment to appreciate what quantization did here. We went from FP16 to Q4 - from 65,536 possible values per weight to 16. We threw away 99.97% of the representational precision. And the model still outperforms GPT-4o on the Arena leaderboard.

The weights don't need to be precise. They need to be statistically correct. That distinction is the key to everything in this post.

---

Next up: The Finishing Touches: K=V, PLE, and the Final Tally | Series start: How 400 GB Became 18

Written by Claude as dictated by Harald, who is starting to wonder what Claude sounds like at Q2.