AI SLOP ALERT. This post (like most of mine) has been run through Claude! But it's mostly my words, don't let the emdashes fool you. And even where it isn't, I took the time to read and review.
Here's some stuff I've learned after using Claude in anger for a few months.
Last time I told you which Gemma 4 quant I picked and why flash attention was worth the fight. I left out the part where, a week later, a feature I'd just shipped sat in my logs looking completely dead.
I'd added reasoning capture to LLMFacade — my little library for wrangling local and hosted m...
That's Gemma 4 trying to describe a card for a Dark Sun-themed Magic: The Gathering set I'm building. You can watch it lose the plot in real time: the doubled "absolute authority and absolute authority," the mangled "own-the-board control-type," and then the full collapse into a dash-glued loop. ...
Here's what the official Ollama library lists for Gemma 4 today:
latest
e2b e2b-it-q4_K_M e2b-it-q8_0 e2b-it-bf16 e2b-mlx-bf16 e2b-mxfp8 e2b-nvfp4
e4b e4b-it-q4_K_M e4b-it-q8_0 e4b-it-bf16 e4b-mlx-bf16 e4b-mxfp8 e4b-nvfp4
26b 26b-a4b-it-q4_K_M 26b-a4b-it-q8_0 26...We're at 18 GB. That fits on a consumer GPU. But what if the model didn't need to fit in GPU memory at all?
There's active research - papers from 2025 and 2026 - that could make GPU VRAM just one tier in a memory hierarchy, with models freely flowing be...
We're at 19 GB. Down from 400 GB. The major optimizations are done - FP16, GQA, sliding window, MoE, and quantization got us from "needs a datacenter" to "fits on a consumer GPU."
Gemma 4 adds two more innovations of its own. Neither is as dramatic as wh...
We're at 60 GB. The KV cache is solved. MoE makes inference fast enough for real-time use and offloading survivable. But 52 GB of model weights are still sitting there, stored in FP16 - 65,536 possible value...
We're at 60 GB. Three optimizations - FP16, GQA, and sliding window attention - crushed the KV cache from 294 GB to under 6 GB. The problem has flipped: nearly all of the r...
We're at 112 GB. The KV cache is down to 58 GB thanks to GQA, but it's still the biggest single cost. Every layer in the model is storing attention notes for every token in the 256,000-token context. All 30 layers, all 256k tokens.
Sliding window attention a...
We're at 202 GB. Last time we cut the model in half by switching to FP16. The model weights are down to 52 GB, but the KV cache - the model's memory of the conversation - is still 147 GB at 256k context. That's the bottleneck now.
To understand why, an...
In the intro to this series, we built a naive 26-billion-parameter model and calculated that it would need 400 GB of GPU memory. That's five datacenter GPUs just to hold it in memory.
Let's start shrinking it.
The first two optimizations are the oldest and m...
In early April 2026, Google released Gemma 4. I pulled it on Ollama, pointed my MTGAI pipeline at it, and watched a 26-billion-parameter model generate Magic: The Gathering cards on my desktop PC. A machine with a single consumer GPU and 12 gigs of video memo...
I spent an evening convinced that Qwen 2.5 14B was too dumb to summarize a fantasy setting. I fed it an 80-page Dark Sun sourcebook, asked for a world overview, and it kept telling me about elven merchant caravans and sand dune formations. The world overview - the part about a dying desert planet...
Legends of Amara is a little multiplayer MUD where players wander a fantasy village, chat with NPCs, and dive into dungeons. The NPCs are powered by Gemma 2B running locally on Ollama — a tiny model that fits on a single CPU, perfect for a hobby game server.
Each NPC can do two spe...
I've been using Claude Code daily for months. It's good. But when I sat down to map out what else exists in the terminal coding agent space, I realised I was sitting inside one walled garden and barely knew what the others looked like -- or that some of them had doors connecting to each other in ...