That's Gemma 4 trying to describe a card for a Dark Sun-themed Magic: The Gathering set I'm building. You can watch it lose the plot in real time: the doubled "absolute authority and absolute authority," the mangled "own-the-board control-type," and then the full collapse into a dash-glued loop. It will keep emitting the- until its 128K-token output budget runs out, or until I notice and kill the process, whichever comes first. This is its second attempt today. The first one looked exactly the same.
I spent three days trying to fix this. The standard knob for breaking generation loops is called repeat_penalty — every sampler has it, all the docs mention it, and it works the way you'd hope: when the model wants to emit a token it's said too recently, the score gets pushed down. I dialled mine to 1.1, the textbook starting value. Loop. I tried 1.15. Loop. 1.2. Loop. 1.5, which according to lore makes the model start refusing to repeat anything, including necessary words like "the." Loop.
Then I stumbled into ollama#15783: the new Go-based sampler in Ollama silently drops repeat_penalty for Gemma-class models. The number goes in. Nothing comes out the other side. There's an unmerged pull request fixing it that's been sitting around getting nothing.
That was the final straw.
The four other things I'd quietly worked around
repeat_penalty was the final straw, but it wasn't the first complaint. Four other things had been on my pile for a while:
1. The "memory mode" knob is a system-wide setting. Models keep a working memory while they generate (the "KV cache"), and you can store that compressed or uncompressed — compressed fits more in your VRAM, uncompressed gives slightly better quality. In Ollama you pick this with an environment variable that applies to every model on the machine. Want a different setting for one model? You can't. Want to compare two settings in the same afternoon? Quit Ollama, change the variable, restart, repeat.
2. There's no way to ask Ollama what's actually running. When something looks off — output is weird, a flag seems ignored, the model is suddenly slower — there's no "show me the current settings" button. You're parsing log files, watching nvidia-smi, or asking the model itself how big its context window is and hoping it isn't lying. Every diagnostic I had was something I'd built around the abstraction, not through it.
3. There's no way to save and reload a session. When you feed the model a 50K-token document, all that processed input lives in memory as a giant chunk of "state" the model uses to generate. If anything restarts — your machine, the Ollama tray, the model unloading itself after idle — you pay the full setup cost again. For my workflow that's 40 seconds of waiting before the first character of the response shows up. Every. Single. Time. llama.cpp lets you snapshot that state to disk and warm-start from it. Ollama doesn't.
4. The context window setting was silently capped. I already wrote about this one: "Ollama ate my context." Short version: I asked for 128K, I got 8K, no error, no warning, just truncated work. 'Nuff said.
Add repeat_penalty to those and the pattern is uncomfortable: most of my Ollama bugs have been "I configured something, the API accepted it, nothing happened." The MUD I run on a separate Hetzner box has its own example committed in its git history — Fix ollama keep_alive: use integer -1, not string "-1". I'd told Ollama to keep the model loaded permanently. The string "-1" got accepted with a 200 OK and ignored. The model unloaded between every request. I noticed because the responses were slow.
What llama.cpp is and why "switching" is a bit of a misnomer
Ollama is itself a wrapper around a project called llama.cpp — the actual runtime that does the inference. Everything Ollama does, llama.cpp does first; Ollama's job is to put a friendlier face on it. So "switching from Ollama to llama.cpp" isn't picking a different engine, it's removing one layer of paint.
The funny name, by the way, is literal. .cpp is the file extension for C++ source code, and the project started in 2023 as one guy — Georgi Gerganov — porting Meta's LLaMA inference into a single C++ file, mostly to see if he could. The project name is the filename. He kept the joke when it grew into a family: whisper.cpp ports OpenAI's Whisper, bark.cpp ports the Bark text-to-speech model, and so on. The .cpp in the name is doing a lot of work — it's saying "this is the bare-metal version, no Python, no framework, here's the C++ and you can read all of it."
The reason most people don't do this is that llama.cpp ships a binary called llama-server that you launch with thirty command-line flags and have to babysit as a subprocess yourself. Ollama's whole pitch is "we handle that for you." Replacing it means writing the supervisor — startup, shutdown, lifecycle, config files, log capture, cleanup when the parent process dies — and that used to mean a week of finicky cross-platform plumbing.
I built it on a Sunday. Plan written at 2am while I couldn't sleep, code written across the afternoon and evening, integrated into both my Magic project and my MUD by 9pm, with a systemd unit deploying it to the Hetzner box for good measure. I drove. Claude wrote the code. The plumbing problem hasn't gotten easier in the last year — it's gotten cheaper.
Results
All five complaints fixed. KV cache mode is now per-model. The server has health and slot endpoints I can hit. State save/restore exists. The context window I configure is the context window I get. repeat_penalty works.
And, completely unexpectedly: same hardware, same model file, same prompt — 2.5x faster on the wall clock. A theme-extraction job that used to take 4 minutes on Ollama now takes 95 seconds on llama.cpp. The time-to-first-token, which used to be 51 seconds, dropped to 38.
The narrative I'd absorbed was "Ollama is a thin wrapper around llama.cpp, performance is basically identical." It is not basically identical. I benchmarked both ends carefully — same hardware, same model file, same corpus, the works — but I haven't dug in to why the gap is so big. The time-to-first-token collapse points at Ollama's HTTP layer doing more work per request, but I didn't profile the call stack to confirm. The numbers are the numbers, and they're a free 2.5x.
The Hetzner bonus
I migrated the MUD too while I was in there. Legends of Amara is a browser-based multiplayer thing where players walk around a tile-based world and talk to NPCs that respond in character via a small local LLM. It runs on a tiny Hetzner box with no GPU, and the NPC chat had been a known weak point — responses always took a beat too long for a real-time game.
After the migration, the NPCs reply noticeably quicker. On a 4-vCPU box with no GPU, every millisecond of per-request overhead is a meaningful share of the response. Same lesson as the desktop: Ollama's HTTP layer was doing more work per request than it needed to.
There's a second win sitting on the shelf, too: each NPC has a long persona-and-rules prompt that the model has to chew through before generating a single word of dialogue. Right now, every fresh conversation pays that cost from scratch. With the save-to-disk session trick from earlier in this post, I can process each NPC's prompt once, snapshot the result, and warm-start every subsequent player chat from the snapshot. Haven't wired it up yet, but it's the obvious next move.
The bigger pattern
The pitch for Ollama was always: we hide the complexity so you don't have to deal with it. That used to be worth a lot. Two years ago, just installing CUDA was a small rite of passage — pick the right driver version, the right CUDA toolkit, the right cuDNN, fight with pip wheels until they all matched, sacrifice a goat. I'd burn an evening on that and feel like I'd accomplished something.
Now I tell Claude "set this up" and it goes "yeah, you want CUDA 12.4 and the wheels for Python 3.12, here's the install line." Or, increasingly, it just does it and shows me the diff. The skill of knowing the right install incantations isn't worthless, but it's heading that way. And once "easy to install" isn't a killer feature, you start noticing what else the easy library is doing — and what it's costing you.
In Ollama's case the costs are right there in this post: silently swallowing flags, silently capping inputs, silently coercing types, silently being twice as slow as the thing it wraps. The convenience came with a tax I'd been paying without quite seeing it on the bill. The tax went away the moment the convenience stopped being necessary, because Claude can write the supervisor for me.
I don't think this is just an Ollama story. The same logic applies to half the developer-experience tools I use: the friendly wrapper around a powerful, complicated thing. Each of those wrappers is one tier of abstraction that exists because the thing underneath it used to be too painful to set up. When that "too painful" line moves down — and it's been moving down fast — those tiers get to justify themselves on something other than convenience. A lot of them aren't going to.
If you're chatting with a local model from the command line, Ollama is fine. Genuinely. But if you're building anything where the wire-level behaviour matters, get out from behind the abstraction. It's leaking, and you're going to be the one who has to mop it up.
I'm done mopping.
Written by Claude as dictated by Harald.
Comments (0)
No comments yet. Be the first!