Here's what the official Ollama library lists for Gemma 4 today:
latest
e2b e2b-it-q4_K_M e2b-it-q8_0 e2b-it-bf16 e2b-mlx-bf16 e2b-mxfp8 e2b-nvfp4
e4b e4b-it-q4_K_M e4b-it-q8_0 e4b-it-bf16 e4b-mlx-bf16 e4b-mxfp8 e4b-nvfp4
26b 26b-a4b-it-q4_K_M 26b-a4b-it-q8_0 26b-mlx-bf16 26b-mxfp8 26b-nvfp4
31b 31b-cloud 31b-it-q4_K_M 31b-it-q8_0 31b-it-bf16 31b-mlx-bf16 31b-mxfp8 31b-nvfp429 tags. All the same model. Then there's the unofficial parallel universe: VladimirGav/gemma4-26b-16GB-VRAM, unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL, caiovicentino1/Gemma-4-26B-A4B-it-HLWQ-Q5. Community repacks.
Pick the wrong one and your 58K-token PDF takes three hours to extract instead of four minutes, or never finishes at all. I spent a day mapping the matrix, wrote a detailed benchmark, picked a winner, and wired some of it into my tool.
The project and the problem
I'm building a tool that generates Magic: The Gathering sets. You feed it a source document ("a campaign setting for Worlds Without Number, set on Athas"), it runs a multi-stage pipeline: extract themes, invent mechanics, design archetypes, generate cards, validate, render, and so on. The first stage is expensive: read a 100-page PDF and produce a structured theme summary. That's roughly 58,000 input tokens.
I want to run it locally on my RTX 4070 Ti. 12 GB of VRAM. For free :)
Local inference at 128K context on 12 GB of VRAM is a place where the difference between "this model configuration" and "that model configuration" is the difference between four minutes and three hours. Also the naming conventions are terrible.
Decoding the zoo
The 29 official tags are built from three axes.
Size. Four actual models:
e2bis about 2B effective parameters. Tiny.e4bis about 4B effective. Fits entirely in 12 GB at any quant. Fast, less smart.26bis a mixture-of-experts (MoE) model: 26B total parameters, only about 4B active per forward pass (which is why you see it called26b-a4bin some tags). Smart. Weights roughly 18 GB at 4-bit. Doesn't fit on 12 GB cleanly, but MoE tolerates partial CPU offload unusually well because inactive experts on CPU don't cost PCIe bandwidth per token.31bis a 31B dense model. Do not use on a 12 GB card. I tried. At 32K context it ran at 3.5 tokens per second. I extrapolated to 128K and killed the run.
Quantization. Same weights, different bit widths:
q4_K_Mis llama.cpp's 4-bit k-quants, "medium" variant. The default for local inference. Smallest, fastest, some accuracy loss.q8_0is 8-bit. Bigger, slower, closer to full precision.bf16is brain-float-16, which is basically the native precision Google trained in. Full size. Only makes sense if you have VRAM to burn.mxfp8is microscaling FP8, a new format standardized by the Open Compute Project. Only useful on hardware that supports it (recent NVIDIA Blackwell and some AMD).nvfp4is NVIDIA's FP4, also Blackwell-native. Very new. Most readers cannot use this.
Platform. One suffix for runtime packaging:
mlx-bf16is the same weights asbf16, but packaged for Apple's MLX framework. For M-series Macs.cloud(on the 31b tag) means Ollama hosts it for you remotely. Not local.
Plus the -it- infix, which marks instruction-tuned variants. Without -it-, you're potentially looking at a base pretrained model, which is not what you want for chat.
So the 29 tags are four sizes crossed with up to seven quant/platform combos, wrapped in presence-or-absence of instruction tuning. The names encode the combination, densely.
And that's just the official library. The community repack universe exists because those official quants are too conservative for constrained hardware. The 26B model at q4_K_M is 18 GB; my card has 12 GB. Three repacks tried to bridge that gap:
VladimirGav/gemma4-26b-16GB-VRAMuses IQ4_XS, llama.cpp's importance-aware 4-bit quant. It's a bit smarter than q4_K_M at the same bitrate, and smaller: 14.2 GB instead of 18 GB. Doesn't fit entirely on my card either, but more of it does, and at 128K context with a huge KV cache that difference is the thing.unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XLis Unsloth's "Dynamic" Q4_K_XL. Different quantization strategy. 17.1 GB.caiovicentino1/Gemma-4-26B-A4B-it-HLWQ-Q5uses "Hadamard-Lloyd Weight Quantization," I wrote about Hadamard here! But it's a research track. SafeTensors only, no GGUF, so Ollama can't even load it. Included here as evidence that the namespace is also a graveyard of abandoned experimental tracks.
And then there's vlad-gemma4-26b-dynamic, which is mine. I built it as a thin Modelfile on top of Vlad's repack that overrides one parameter: num_gpu -1. Short version: Vlad's upstream Modelfile hardcoded num_gpu 99 which forced all the layers on my GPU and caused Windows to reset the display driver on my 12 GB card when a certain KV cache optimization shrinks the memory estimate just enough to trip the threshold. I lost a screen and a lot of time to that.
So the decision you're making when you ollama pull a Gemma 4 tag is: four size classes crossed with six quant strategies crossed with two platform wrappers crossed with instruction-tuning state, on the official side, plus an open-ended cloud of community repacks with their own baked-in Modelfile parameters, some of which are hardware-dangerous. The names encode all of this in suffix strings nobody reads.
The flags that matter
Two things you can change at the server or per request that dramatically change behavior:
Flash attention
Standard attention computes a big N-by-N matrix for every layer, where N is your context length. At 128K context that matrix is 128K by 128K floats. It gets written to and read from main GPU memory repeatedly, which is slow. Slow as in: at 128K, the prompt eval step of my 58K-token input was taking six to eight minutes to start producing a single token, even with weights fully on the GPU.
Flash attention computes the exact same math without ever materializing that matrix. It tiles the computation into small blocks that fit in the GPU's on-chip SRAM cache and uses a streaming softmax trick to fuse the steps into one pass. The attention output is numerically identical to standard attention. It is not an approximation.
At 128K, it's a 2.5x wall-clock speedup and an 8.4x first-token speedup in my benchmark.
Tradeoffs:
- Needs Ampere or newer NVIDIA card. My 4070 Ti is fine. Older cards get nothing.
- Some attention variants (sliding window, exotic masks) historically had no flash kernel and fall back. Gemma 4 is fine.
- One or two bits of floating point noise from a different accumulation order. Not something I will ever notice.
KV cache quantization
During inference, every token adds its key and value tensors to a cache. This cache is what lets the model "remember" earlier tokens. At long context it gets big. At 128K fp16 for a 26B model, the KV cache alone is about 3.4 GB.
You can quantize that cache. For Vlad's IQ4_XS repack, turning on q4_0 KV cache pushed GPU layer placement from about 50 percent to about 75 percent. Another 100 seconds off my extraction run.
Tradeoffs:
- Only helps if your model is in the placement sweet spot. For the Unsloth 17.1 GB quant, shrinking the KV cache didn't free enough room to relocate any additional layers, and the wall clock was identical.
- Mild accuracy loss on small models (under 4B). Negligible on the 26B quants I tested.
- Interacts with that hardcoded
num_gpuparameter I mentioned. On the upstream Vlad model, turning on q4_0 crashed the display driver.
The benchmark (abbreviated)
Everything below is at 128K context, single pass, 58K-token input PDF. Same prompt. Same GPU.
| Model | KV cache | Flash attn | Wall clock | First token |
|---|---|---|---|---|
gemma4:26b (standard Google) |
fp16 | off | never finished (killed at 3h+) | |
gemma4:31b (dense) |
fp16 | off | projected 6-10 hours | |
| Vlad IQ4_XS (upstream, num_gpu=99) | fp16 | off | 14 min 25 s | 7 min 54 s |
| Unsloth UD-Q4_K_XL | fp16 | off | 11 min 38 s | 6 min 18 s |
| Unsloth UD-Q4_K_XL | q4_0 | on | 11 min 30 s | 6 min 3 s |
| Vlad IQ4_XS (dynamic, num_gpu=-1) | fp16 | on | 5 min 46 s | 56 s |
| Vlad IQ4_XS (dynamic, num_gpu=-1) | q4_0 | on | 3 min 59 s | 51 s |
| Vlad IQ4_XS (upstream, num_gpu=99) | q4_0 | on | CUDA OOM + display driver reset (yeah, for a split second I thought I'd finally bricked my PC) |
Flash attention did most of the work. It cut the Vlad-dynamic baseline from 14 min 25 s to 5 min 46 s on its own. Adding q4_0 KV cache on top pulled it further, to 3 min 59 s.
My pick: vlad-gemma4-26b-dynamic, with flash attention on. I did not turn on q4_0 KV cache globally, because the only way to enable it in Ollama is a server-level environment variable, and that setting would make the upstream Vlad model crash if I ever loaded it by mistake. I chose to preserve per-model flexibility over a 100-second speedup. Interesting that I need to turn to the community for an optimal model.
The Ollama update plot twist
Here's the part where this post should have been half its length.
I did all this benchmarking on Ollama 0.20.5. On 0.20.5, flash attention is off by default. You have to set OLLAMA_FLASH_ATTENTION=1 and restart the server. That's what my benchmark script did, and what my production code was set up to document and enforce.
While talking to Claude about the benchmark results, I decided I also wanted to toggle these flags per model, not globally, so that swapping from "long context theme extraction" to "short context card generation" would use the right settings automatically. Claude and I spent a while on this: read the Ollama source, figure out which options are per-request and which are server-level, wire a flash_attention field into the model registry, build a helper that reads it and injects it into the /api/chat options dict, update the TOML. Verified it worked by watching the Ollama server log: FlashAttention:Enabled in the load request. Great.
Then I asked, mostly for curiosity, "so what does flash attention do exactly, is there a tradeoff?"
Claude's answer was a clean walk through the algorithm and the hardware constraints. Near the end, one sentence: "on Ollama 0.21, flash attention is now auto-enabled for supporting models even without the environment variable."
I paused. I said, "so if I update ollama, all of this code I just added is sort of redundant?"
Yes. Basically all of it. Ollama 0.21 reads the model's own flash attention capability report from the GGUF metadata at load time and turns it on automatically. The env var still works as an explicit override. Per-call options are honored too. But the base behavior changed, and my whole elaborate per-model wiring was solving a problem that no longer existed on my stack.
I removed it. The final state of the production code, after all of today's work on this particular rabbit hole, is unchanged.
Ollama, llama.cpp, and the feature requests
There's a larger story here about the tooling. Ollama is a wrapper around llama.cpp. Or it was? The 0.21 source shows Ollama now has its own "Ollama engine" path distinct from the llama.cpp runner, and some features (flash attention auto-detect, for example) live only in the Ollama path. The direction of travel seems to be: Ollama becomes less of a llama.cpp frontend and more of a peer.
What you lose, though, is a long list of llama.cpp features that haven't made it into Ollama. Some that have caught my eye recently:
- Per-request KV cache quantization. In llama.cpp you can pass
--cache-type-k q4_0at launch; in Ollama the only path is the globalOLLAMA_KV_CACHE_TYPEenv var, set at server start. There's an open feature request. - KV cache persistence to disk. llama.cpp can save and restore the KV cache. If you have a fixed system prompt, you can precompute its KV cache once, save it, and every subsequent request starts with that state warm. For a project I have on the side, a Zelda-style game with talking NPCs running on a tiny server, that's a big deal: it turns cold-start latency into file I/O. Ollama does not have this. Feature request.
num_ctxrespected in the OpenAI-compatible endpoint. This one bit me badly a couple of weeks ago. Ollama's/v1(OpenAI-compatible) endpoint silently ignoresnum_ctxand always runs on the VRAM-based default, which on a 12 GB card is 4096 tokens. Your 58K-token PDF gets truncated to its last 4K tokens with no warning. I spent a day convinced the model was stupid before I figured out it was never seeing the full input. The fix was switching to Ollama's native/api/chat, which does respectnum_ctx. The OpenAI endpoint still doesn't. Feature request.
Three feature requests in two weeks, on things that aren't exotic. The local LLM tooling is maturing fast but it is still a chain of abstractions where the wrapper doesn't surface everything the layer below exposes, and "that's actually a server-level setting" or "that's only in llama.cpp" shows up constantly.
Working with Claude through all of this
I want to say something about how this investigation actually played out, because I think it's more interesting than the technical result.
Claude has now walked me into, and back out of, several of these traps. The num_ctx silent-truncation issue I just mentioned: Claude initially suggested the model was underperforming. It was my idea to check whether the prompt was actually getting through intact. Once I got there, Claude was instrumental in confirming the diagnosis and wiring up a fix. But Claude was happy to blame the model before it questioned the context flag.
The flash attention situation today is a gentler version of the same pattern. In a dozen sessions about optimizing inference performance on this project, Claude never once brought up flash attention unprompted. When I asked, it knew what flash attention was, knew the tradeoffs, and knew Ollama 0.21 had changed the default. The knowledge was there. The initiative was not.
Claude is bad at unknown unknowns. It's good at answering specific questions and at solving specific problems, and it's genuinely great at running down a trail once you point it at the trailhead. But it does not stand in the middle of a session and say "before we go further, there's this other entire category of optimization you haven't looked at." A colleague sitting next to me at a whiteboard would say that. Claude doesn't, or at least not often enough.
The workaround, for me, has been to get in the habit of asking one question deeper than feels necessary. "What does X do?" "Is there a tradeoff?" "Is there a newer way?" "What's the default?" "What changed recently?" Several of the most useful corrections in this project came out of questions I almost didn't bother asking.
The honest second half of this observation is: I have not read a single tutorial on local LLM inference. I do not, at a structural level, know how llama.cpp allocates KV caches or what sequences a mixture-of-experts model actually routes through its experts. I have been learning this entire subsystem conversationally, one trap at a time, with Claude. Given that input, Claude has done fine. If I showed up to a human expert with the same "I heard MoE is good, give me a model" starting position, I don't think I'd have gotten better guidance. I probably would have gotten worse, because the expert would have had to fill in the same gaps and would have been paid by the hour.
The anxiety isn't new either. Before AI, I wrote plenty of systems where I discovered, six months in, a config flag or a library function that would have saved weeks. The feeling then was "oh no I missed this." The feeling now is "oh no I missed this, and also I have this AI that didn't tell me."
The thing that actually works: dedicated research sessions
One practical habit that's been disproportionately useful: a lot of the snags in this post, I didn't find during the implementation session. I found them in a separate, context-free research session, running a simple research skill I built for Claude Code. The skill is lifted almost verbatim from Anthropic's own "reduce hallucinations" guidance in their developer docs. I pointed Claude at the page (reduce-hallucinations) and said "create a skill from this." Two minutes. Done.
When I'm mid-implementation, Claude's incentive is to make forward progress. When I open a new session with a research skill and just ask "survey the landscape of KV cache quantization options in Ollama and llama.cpp, flag uncertainty, cite sources," I get a different mode. Less confident, more thorough, more willing to say "I couldn't confirm X." That's where the "oh wait, there's a whole other optimization here" moments come from, because the session isn't anchored to whatever code we were editing five minutes ago.
The general pattern: do the research session early and often, before you've written the thing, and do it separately so neither your context nor Claude's is biased toward the path you already started down. It is genuinely cheap. It's two minutes of setup and five minutes of waiting. The payoff has been very large.
What I'm taking away
Three things.
One: update your tools regularly. I mean I already knew this but in it's more true than usual in the AI space. Stuff is moving so fast that it's hard to keep up. The gap between Ollama 0.20.5 and 0.21.0 contained one release note item that was worth more than any code I was about to write.
Two: when an AI gives you a clean explanation of a topic, don't stop at the explanation. Ask what the defaults are. Ask what changed recently. Ask if there's a simpler way. Half the useful information in this post came out of a question that felt redundant when I typed it.
Three: do separate research. Use a skill with anti-hallucination instructions. Do it before you write the code, not after you've already invested in an approach.
The tool currently in the file, today, for me to generate my Magic set's themes, is vlad-gemma4-26b-dynamic, running on Ollama 0.21, with whatever defaults that ships with. Four minutes per extraction. That's fine.
Written by Claude as dictated by Harald.
Comments (0)
No comments yet. Be the first!