More adventures in local LLM land

Last time I told you which Gemma 4 quant I picked and why flash attention was worth the fight. I left out the part where, a week later, a feature I'd just shipped sat in my logs looking completely dead.

I'd added reasoning capture to LLMFacade — my little library for wrangling local and hosted models behind one interface. When a model thinks out loud, the chain-of-thought now lands in the logs, collapsible, with a token count beside it. I built it, tested it, shipped it, and opened the logs of my Magic: The Gathering mechanic-reviewer to admire all that lovely reasoning.

Forty-eight review turns. Every single one: thinking: null.

the feature that was dead on arrival

The reviewer is a council of local models that judge custom MTG card mechanics — each one reads a mechanic, picks it apart against six standards, and submits a verdict through a tool call. Exactly the workload where you want to see why a model rejected something.

And there was nothing to see. Empty thinking field, every turn.

The obvious first suspect, when a feature you just wrote produces nothing, is the feature you just wrote.

suspect one: my own code

So I traced it. Same model, same prompt, four different ways of talking to it:

• raw HTTP to llama-server
• the OpenAI SDK (which my llama.cpp provider uses as its wire client)
• LLMFacade pointed at an external server
• LLMFacade in managed mode, where it launches the server itself

A plain "what's 17 times 23, think step by step" went in. All four came back with 700+ characters of chain-of-thought, captured correctly, rendered in the log.

The capture pipeline worked. It's a rare and faintly disorienting thing to rule out your own code this early in a hunt.

suspect two: the flag everyone blames

llama.cpp has a flag, --jinja, that tells the server to render the model's own embedded chat template instead of a built-in guess. My managed mode wasn't passing it. This smelled exactly right — wrong template handling, lost reasoning.

I added it. Tested again. Reasoning came through with --jinja and without it, identically.

Wrong again. Two suspects down, 48 empty logs to go.

oh. it's the tool.

The difference between my passing tests and my failing logs wasn't the transport, and it wasn't the flag. It was the tool.

Every review turn hands the model a submit_mechanic_review function and asks for its verdict that way. And the instant there's a tool on the table, the model skips thinking entirely and jumps straight to the call. I reproduced it with the exact reviewer prompt: empty text, one clean tool call, zero reasoning. Every time.

So the question flips: is this thing even a thinking model, or did I imagine those 700 characters?

It thinks. Toolless prompts produce piles of chain-of-thought. It just refuses to do it when a tool is in the room.

the template was lying about its age

My quant was a community repack — vlad-gemma4-26b-dynamic, picked (as you'll recall) for fitting nicely in 12 GB. When llama-server loads it, it prints this:

common_chat_try_specialized_template: detected an outdated gemma4 chat
template, applying compatibility workarounds. Consider updating to the
official template.

Outdated template. Unsloth had shipped a fix in April that, by their notes, specifically improved tool-calling. So I went to grab a fresh quant — gemma-4-26B-A4B-it-UD-IQ4_XS, 13.6 GB.

Which is where local LLM land threw in a bonus sidequest. hf download hung at zero bytes for two minutes. The repo is Xet-backed, the hf_xet accelerator wasn't installed, and the fallback just... sat there, lock file and all. The fix was to stop being clever:

curl -L -C - -o gemma-4-26B-A4B-it-UD-IQ4_XS.gguf \
  "https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/resolve/main/gemma-4-26B-A4B-it-UD-IQ4_XS.gguf"

31 MB/s, no drama. Even the download tools have quirks down here.

reading a 13 GB file without reading it

Before relaunching anything, I wanted to know what was actually in these templates. Turns out you can pull a GGUF's embedded chat template straight out of the file header — no weights, no inference, just metadata. About ninety lines of standard library:

import struct
with open(path, "rb") as f:
    assert f.read(4) == b"GGUF"
    struct.unpack("<I", f.read(4))                  # version
    _tensors, kv_count = struct.unpack("<QQ", f.read(16))
    for _ in range(kv_count):
        key = read_str(f)
        (vtype,) = struct.unpack("<I", f.read(4))
        val = read_value(f, vtype)                  # captures strings, skips the rest
        if "chat_template" in key:
            templates[key] = val

Run it across vlad's template and unsloth's, and the interesting bit isn't the version difference — both support thinking. It's how you turn it on:

signatures present: {'enable_thinking': 5, 'strip_thinking': ..., 'tool_calls': ...}

Gemma 4 controls thinking through an enable_thinking template flag — the same convention Qwen3 uses — not some magic token you sprinkle into the prompt. That matters twice: there's a real, detectable lever, and a library could sniff the template at load time and wire that lever up on its own.

(The other thing the diff showed: vlad's template mentioned tool_calls twice; unsloth's, five times. The "fix" was entirely about how thinking and tool calls coexist. Foreshadowing.)

the matrix

Armed with the actual lever — --jinja so the template runs, plus chat_template_kwargs={"enable_thinking": ...} — I ran the real combinations against the exact reviewer prompt:

thinking	tool_choice	reasoning	tool call
off	auto or forced	—	clean call
on	auto	yes	sometimes
on	forced	23,635 chars	none

Read the top two rows and the shape is clear: this model will think, or it'll call your tool, but it does not want to do both. Thinking on, it reasons at length and then answers in prose, ignoring the function. Thinking off, the tool call snaps into place instantly.

This isn't just my quant being difficult. Fireworks, writing about Qwen 3, put it plainly: "open-source LLMs forced a choice between showing chain of thought or calling tools deterministically." It's a known tradeoff. Newer architectures are built to escape it; this one hasn't.

forcing made it worse

So I did the thing that feels obviously correct: if the model won't reliably call the tool on its own, force it. Set tool_choice to require the function. Surely that guarantees the call — and maybe lets it think first too.

Forced, with thinking on: 23,635 characters of "reasoning," finish_reason: stop, and no tool call at all.

This made no sense until I found llama.cpp issue #20809. When the model is in thinking mode and you force a tool, the tool-call markup gets parsed into the reasoning field instead of the tool-calls field:

"reasoning_content": "<tool_call>\n{\"name\": \"...\", \"arguments\": {...}}\n</tool_call>"
"tool_calls": null

My tool call wasn't missing. It was sitting inside the 23k-character reasoning blob, unparsed, with the wrong finish reason. A maintainer's take in a related thread (#12204) is roughly: you can't grammar-constrain a reasoning model without wrecking it — the constraint stops it from emitting the reasoning tokens it was trained to lean on.

So "force" isn't more control. It's a parser trap that eats your output and files it under the wrong key.

what actually works

--jinja, enable_thinking=true, tool_choice=auto (never forced), and a retry if no tool call comes back. You get the reasoning and the structured call — just not on every roll of the dice, so you loop until it cooperates.

Anticlimactic? A little. But it's honest about what this model is: a thinking model that treats "call a tool" as one more thing it might decide to do once it's finished thinking.

down here, it's quirks all the way down

A few things this cost me that are worth saying out loud.

The bug is almost never where you first look. I burned two solid hypotheses — my code, then --jinja — before the real culprit surfaced: the tool, then the template, then a parser routing bug, stacked three deep. Local LLM land is layers — model, quant, chat template, inference server, transport SDK — and every one of them has its own quirks.

Leaving Ollama for llama.cpp didn't get me away from bugs. Ollama wraps llama.cpp; they share the exact parsing layer this bug lives in. I didn't escape it — I just got closer to the logs that let me see it. (Related: llama.cpp doesn't auto-update. It's a binary you build or download, so a stale binary is a stale bug. This one's on a "rebuild and re-test occasionally" reminder now.)

And the whole reason LLMFacade exists is to eat all of this. The goal is that I write thinking=True and never have to remember what chat_template_kwargs is or which field a forced tool call gets misfiled into. There's a plan to fold the auto-detection in — sniff the template, pick the lever, hide the retry. That's the next adventure.

For now, the log shows reasoning again. It only took ruling out everything that wasn't the problem first.

Written by Claude as dictated by Harald.