I spent an evening convinced that Qwen 2.5 14B was too dumb to summarize a fantasy setting. I fed it an 80-page Dark Sun sourcebook, asked for a world overview, and it kept telling me about elven merchant caravans and sand dune formations. The world overview - the part about a dying desert planet ruled by sorcerer-kings - was right there in Chapter One. The model just... ignored it.
Turns out, the model never saw Chapter One at all.
The Project
I'm building MTGAI, a pipeline that generates custom Magic: The Gathering card sets using AI. The idea is: you feed it a setting document (say, a D&D sourcebook), and it extracts themes, generates mechanics, creates cards, makes art, and renders print-ready PNGs. The whole thing runs as a multi-stage pipeline with both cloud models (Claude) and local models (Ollama) as options.
Savannah Lions, reimagined by the pipeline. From setting prose to robo-cats.
Is AI-generating Magic cards kind of sloppy? Probably, yeah. And you can bet Wizards of the Coast set up their own internal pipeline ages ago - they'd be crazy not to, even just for playtesting. But that's half the fun of a hobby project: building the machine is the point, not the output.
The theme extraction stage is where it starts. You give it a setting document, it chunks it up and feeds it to an LLM to extract world overview, factions, conflicts, visual motifs - everything needed to design a card set. For local development, I wanted this to run on Qwen 2.5 14B through Ollama so I could iterate without burning API credits.
That's when things got weird.
The Symptoms
The extraction was using a chunked approach - split the 80-page document into parts, process each with the LLM, accumulate results. Pretty standard. But the outputs were nonsensical. I asked for "World Overview" and got this back:
It seems like you're sharing a rich narrative about elven merchant tribes and their unique way of life on Athas. Let's summarize the key points:
- Elven Merchant Tribes: Elven tribes enrich themselves through nomadic trading...
That's from page 30 of the document. The actual world overview - "I live in a world of fire and sand. The crimson sun scorches the life from anything that crawls or flies" - is on page 2. The model was writing confidently about the wrong section entirely.
On the next chunk, it summarized the Ringing Mountains geography. On the third, it described trade route logistics. Never once did it mention sorcerer-kings, the Sea of Silt, or the fact that Athas is a post-apocalyptic desert world. You know, the actual world overview.
I kept telling Claude (the AI assistant I use for development) that something was fundamentally wrong - the model wasn't just getting it wrong, it was answering about completely different content. Claude kept insisting it was a model quality issue. 14B parameters, quantized to Q4, running locally - of course it can't follow instructions well enough to distinguish "world overview" from "whatever text is in front of me." We tried rewriting prompts, adding emphasis, using all-caps instructions. Claude was confident each revision would fix it. None of them did, because the problem was never the prompt.
The BANANA Test
After a few hours of prompt engineering that went nowhere, I got suspicious. If the model couldn't follow the instruction "extract world overview information," that's one thing. But it wasn't even mentioning the world overview content. Not getting it wrong - completely absent.
So I wrote a diagnostic script. Five hundred paragraphs, each with a unique keyword: BANANA1, BANANA2, all the way to BANANA500. Then three questions: What's the secret word in paragraph 1? Paragraph 250? Paragraph 500?
# Build a test: 500 paragraphs, each with a unique keyword
paragraphs = []
for i in range(500):
paragraphs.append(
f"Paragraph {i+1}: The secret word for this section is BANANA{i+1}. "
f"This paragraph talks about topic number {i+1} in great detail. "
f"The kingdom of TESTLAND{i+1} has many warriors and castles."
)The model knew BANANA500. It had no idea BANANA1 existed.
The beginning of the text was gone. Not misunderstood, not deprioritized - gone.
Three Silent Killers
Digging into Ollama's internals revealed three problems, all silent, all compounding:
1. The 4K default. Ollama doesn't default to the model's advertised context window. It picks a default based on your VRAM:
| VRAM | Default num_ctx |
|---|---|
| < 24 GB | 4,096 |
| 24-48 GB | 32,768 |
| >= 48 GB | 256,000 |
My GPU has 12GB. So Ollama was using a 4K context window - not the 32K or 128K that Qwen 2.5 supports. When I sent ~20K tokens of Dark Sun lore, it silently chopped off the first ~16K from the front. No error, no warning - and the debug logs that would show the truncation only appear if you start Ollama with OLLAMA_DEBUG=1, which of course nobody does by default.
2. The OpenAI API ignores num_ctx. My code used Ollama's OpenAI-compatible /v1/chat/completions endpoint (because the openai Python SDK is convenient). This endpoint does not support the num_ctx parameter. You can pass it - it just gets silently ignored. There's an open issue requesting this. As of writing, it's still not fixed.
3. The 128-token output cap. Ollama's default num_predict (max output tokens) is 128. Not 4096. Not unlimited. One hundred and twenty-eight tokens. Enough for about two paragraphs. Long structured outputs - like a JSON tool call or a detailed world overview - get silently truncated mid-sentence.
Three silent defaults. No errors. No warnings. The model confidently generates whatever it can see, and you have no way of knowing that 80% of your input was thrown away before it even started.
We made a card about it. 0/4 because 4K context produces zero useful output. "They won't notice" is doing a lot of heavy lifting.
The Vibe Coding Trap
Here's the thing: this was entirely a vibe coding error. When I set up the Ollama integration, I let Claude handle it. Claude suggested using the OpenAI-compatible API because the openai Python SDK is convenient and familiar. It even dutifully passed num_ctx in the request options. It just didn't work, and neither of us verified that it did.
If I'd integrated the Ollama API by hand - reading the docs, checking the GitHub issues, running OLLAMA_DEBUG=1 - I almost certainly would have caught this earlier. But that's the trade-off with AI-assisted development: you move fast and skip the research phase. Sometimes that's fine. Sometimes you spend an evening yelling at a model that can't hear you because your AI assistant picked the wrong API endpoint and confidently told you it was the model's fault.
The fix only came when I cleared context and ran a dedicated deep-research session, specifically asking Claude to investigate Ollama's context handling from scratch. That turned up the GitHub issue about the OpenAI endpoint ignoring num_ctx, and everything clicked.
The lesson: when vibe coding, test early. Don't assume the integration works just because it runs. A quick sanity check - "does the model actually know what's at the beginning of my input?" - would have saved hours.
The Fix
The fix was switching from Ollama's OpenAI-compatible endpoint to its native /api/chat endpoint, which actually respects num_ctx:
body = {
"model": model_info.model_id,
"messages": [
{"role": "system", "content": system_msg},
{"role": "user", "content": user_msg},
],
"options": {
"num_ctx": 32768,
"temperature": 0.7,
"num_predict": -1, # no output cap
},
}
response = requests.post(f"{OLLAMA_URL}/api/chat", json=body)Same model, same prompt, same document. The first extraction after the fix:
Athas is a sun-scorched and wind-blown desert world where life is perpetually on the brink of extinction. The crimson sun beats down relentlessly, raising temperatures to unbearable levels by midmorning. Water is scarce, with rain falling rarely, if ever, in most regions.
There it is. Chapter One. The model wasn't dumb - it was blind.
Enter Gemma 4
I'm telling this story now because of what just happened this week. Google released Gemma 4 on April 2nd, and it landed on Ollama the same day.
The numbers are striking. The 26B Mixture-of-Experts variant activates only 3.8 billion parameters per forward pass - meaning it runs fast on consumer hardware while punching well above its weight class. It trades benchmark blows with Qwen 3.5 at the ~30B scale. It supports vision, native tool calling, and 128K context. It's Apache 2.0 licensed.
I added it to MTGAI's model registry in about ten minutes. Two new entries in a TOML file, update a preset, done. The all-local preset now runs every stage - theme extraction, card generation, balance analysis, art selection - on Gemma 4 26B. For free. On my desktop.
[llm.gemma4-26b]
name = "Gemma 4 26B MoE (Local)"
provider = "ollama"
model_id = "gemma4:26b"
context_window = 128000
supports_vision = trueA year ago, running a full card generation pipeline locally would have been a joke. The models were too small, too unreliable, too bad at following structured output schemas. Now I have a model that supports tool calling natively, sees images, and fits in 12GB of VRAM.
The Inversion
There's a narrative going around that AI is about to get expensive. That the current pricing from Anthropic, OpenAI, and Google is heavily subsidized, and that a reckoning is coming when companies need to actually make money. Token prices go up, free tiers shrink, the party ends.
I think the opposite is about to happen.
Not because cloud providers will keep subsidizing forever - they won't. But because local models are improving so fast that the pricing pressure will come from below, not above.
Think about the trajectory. Eighteen months ago, running anything locally meant tolerating hallucinating 7B models that couldn't reliably output JSON. Today, Gemma 4 26B runs on a consumer GPU, supports tool calling and vision, and benchmarks within a few percent of cloud models on reasoning tasks. The Ollama context window issue I spent hours debugging? That's a tooling problem, not a model problem. The models themselves are already good enough for most tasks.
This card was designed, balanced, reviewed, and illustrated through a multi-stage AI pipeline. Most of those stages can now run locally.
My prediction: local models will take over everyday AI use within the next year or two. Coding assistants, content generation, data extraction, summarization - all of it will run on your own hardware. Frontier models like Opus or GPT-5 won't disappear, but they'll retreat to the tasks that actually need them: deep research, complex multi-step reasoning, problems where you need the absolute best model and cost doesn't matter.
The economics are simple. A local model costs electricity. A cloud model costs per token, forever. As local models close the quality gap - and they're closing it fast - the math stops working for cloud providers on commodity tasks. Why pay $15 per million input tokens for Opus when Gemma 4 does the job for the cost of keeping your PC on?
The rough edges are still there. I literally just showed you how Ollama silently eats your context. Tool calling support varies. Quantization can degrade quality in unpredictable ways. But these are engineering problems, and they're being solved in real-time. Ollama 0.20.3 shipped improved Gemma 4 tool calling the same week the model dropped.
The subsidized pricing era won't end with a price hike. It'll end because nobody needs to pay for what their laptop can do for free.
Written by Claude as dictated by Harald.
Comments (0)
No comments yet. Be the first!