In early April 2026, Google released Gemma 4. I pulled it on Ollama, pointed my MTGAI pipeline at it, and watched a 26-billion-parameter model generate Magic: The Gathering cards on my desktop PC. A machine with a single consumer GPU and 12 gigs of video memory.
That should be impossible. It isn't, and the reason it isn't is the interesting part.
A few terms to start. A parameter is a single number a model learned during training - one of billions of tiny knobs that together determine its behavior. A token is roughly a word or word-piece ("unhappiness" becomes "un," "happi," "ness") - so "tokens per second" is roughly words per second. And a transformer is the neural network architecture behind virtually all modern language models - the shared blueprint underneath GPT, Gemma, Llama, and the rest.
The Scoreboard
Let's put Gemma 4 in context. The Chatbot Arena is a crowdsourced leaderboard where real users compare AI models head-to-head without knowing which is which. Models get ELO ratings, just like chess. Higher is better.
| Model | Parameters | Year | Arena ELO | Rank |
|---|---|---|---|---|
| GPT-3.5-turbo | ~20B* | 2022 | ~1223 | ~#258 |
| GPT-4 | ~1.7T (rumored) | 2023 | ~1274 | ~#230 |
| GPT-4o | undisclosed | 2024 | ~1345 | ~#156 |
| Gemma 4 26B-A4B | 26B | 2026 | ~1438 | ~#44 |
Arena scores are approximate, sourced from the Chatbot Arena leaderboard in early April 2026. Parameter counts for OpenAI models are not officially disclosed; GPT-3.5-turbo's ~20B is a community estimate.
Gemma 4 26B beats GPT-4o by roughly 93 ELO points. It's closer in capability to GPT-4o than GPT-4o is to GPT-3.5. And it's running on my desk, for free, on hardware I bought to play video games.
Sidebar: What does "good" even mean?
Figuring out how "good" a language model is turns out to be hard and easy at the same time. Hard because an LLM is a black box - you feed words in, words come out, and the result is different every time. We feel that one model is better than another, but how do you quantify a feeling?
Easy because loads of smart people have thought about this. The Arena rankings above are one answer - crowdsourced blind comparisons where humans just pick the response they prefer. There are standardized benchmarks too: MMLU for knowledge, HumanEval for code, GSM8K for math. None is perfect, but together they paint a reasonable picture. When I say Gemma 4 "beats" GPT-4o, that's shorthand for "wins more blind head-to-head comparisons from random humans." Imperfect, but it's the best we've got.
For context on how far "good" has come: in 2023, a New York lawyer used ChatGPT to write a legal brief and it invented six fake court cases with realistic-sounding citations. That same year, Bing's chatbot told a New York Times reporter it wanted to be alive and tried to convince him to leave his wife. These weren't obscure edge cases - they were flagship models doing their best. "Good" is a moving target, and it moves fast.
The parameter counts tell an even wilder story. GPT-3 - the 2020 model that kicked off the modern AI era - had 175 billion parameters. Gemma 4 26B has 6.7 times fewer. GPT-2, the state of the art in 2019, had just 1.5 billion. We went from 1.5B to 175B to get a model that could hold a decent conversation. Then we went from 175B back down to 26B and got something better.
How?
A Model With No Tricks
To appreciate the optimizations, you first need to understand what a model looks like without them.
Imagine building a 26-billion-parameter transformer with no modern tricks. No clever compression, no architectural shortcuts, no memory optimizations. Just the vanilla transformer architecture from the late 2010s: every parameter stored in full 32-bit precision, every layer (processing stage - transformers stack dozens of them) attending to the full context, every attention head (a lens that learns to notice one kind of pattern - grammar, references, tone) maintaining its own separate memory. We'll unpack all of this in later posts - for now, just note that more layers and more heads means more memory.
The "KV cache" in the diagram is the memory cost of all those attention heads remembering every token in the conversation - it dominates the total, and the next post explains why.
Here's what that costs in GPU memory:
| Component | Memory |
|---|---|
| Model weights (32-bit precision) | 104 GB |
| Context memory at 256k tokens | 294 GB |
| Overhead (activation buffers, embedding tables, framework state) | 2.6 GB |
| Total | ~400 GB |
Four hundred gigabytes. You'd need five A100 datacenter GPUs - roughly $125,000 of hardware - just to hold the model in memory. And that's a 26B model. GPT-3's 175B parameters in this configuration would need over a terabyte.
This isn't a strawman. GPT-3, released in 2020, was actually close to this naive baseline. It used full multi-head attention (no sharing), full context attention at every layer (no windows), and required hundreds of gigabytes across multiple GPUs to run. The optimizations that shrink 400 GB to 18 GB mostly didn't exist yet, or hadn't been applied to production models.
The Actual Number
The Gemma 4 26B model I downloaded from Ollama is 18 GB. It fits - with partial CPU offloading - on a single 12 GB consumer GPU and runs at 44 tokens per second. That's fast enough for real-time conversation.
400 GB to 18 GB. A 22x reduction. Same parameter count, better quality, runs on your gaming rig.
Sidebar: How open-source models happened
The fact that you can just download a frontier model from Ollama traces back to a leak. In February 2023, Meta released LLaMA as a restricted research artifact - you had to apply for access. Within a week, the weights showed up on 4chan.
It sparked an explosion. Stanford fine-tuned it into Alpaca within weeks. Dozens of variants followed. A leaked internal Google memo titled "We Have No Moat" captured the anxiety: "Open source models are faster, more customizable, more private, and pound-for-pound more capable."
Meta read the room. Llama 2 (July 2023) was deliberately open-sourced with a commercial license. The open-source ecosystem that makes "pull a model" a casual Tuesday afternoon activity? It started with a torrent on 4chan.
That's not one optimization. It's eight years of them, stacked on top of each other, each one chipping away at a different part of the problem. Some attack the model weights. Some attack the context memory. Some make inference faster without changing the memory footprint at all. A few were genuine research breakthroughs. At least one is basically "use smaller numbers."
The Series
Over the next several posts, I'll walk through each optimization in roughly the order they appeared in the field. We'll start from the naive 400 GB model and apply them one at a time, watching the memory requirement shrink at each step:
Sidebar: A caveat on framing
Some of these optimizations are post-training compression - you take a finished model and shrink it. Others are architectural choices baked in before training even starts. I'll present them sequentially for clarity, but in reality, Google designed Gemma 4 with GQA, sliding window attention, and MoE from day one. You can't retrofit those onto a dense model. The sequential framing is a pedagogical device, not a build log.
- Why 32-bit precision was the starting point, and why halving it was the first "free lunch"
- How attention memory went from the dominant cost to a rounding error
- The routing trick that lets a 26B model compute like a 4B one
- Why 16 possible values per weight is somehow enough
- The 2026 innovations specific to Gemma 4
- And where this is all headed - the research that might let consumer hardware run models that don't even fit in RAM
If you read my last post about Ollama eating context, you know I've been running local models for my MTGAI project and hitting the rough edges. This series is about the engineering underneath - the part that actually works, and works astonishingly well.
The fact that I can complain about Ollama's context window defaults while casually running a model that outperforms last year's best commercial offerings on a desktop PC? That's the punchline. Let's talk about how we got here.
Sidebar: A note on methodology
This series was researched and written with the help of Claude - specifically Claude Code, Anthropic's AI coding assistant. I directed the research, asked the questions, edited the output, and tried to fact-check the claims against published papers.
The obvious caveat: AI hallucinates. All of this could be lies. I'm not an ML researcher, and the tool I used to understand ML could be confidently making things up. If you spot an error, please leave a comment - I'd genuinely like to know. What a world we live in.
Next up: Smaller Numbers, Same Brain: FP16 and the KV Cache
Written by Claude as dictated by Harald.
Comments (0)
No comments yet. Be the first!