Models bigger than memory

We're at 18 GB. That fits on a consumer GPU. But what if the model didn't need to fit in GPU memory at all?

There's active research - papers from 2025 and 2026 - that could make GPU VRAM just one tier in a memory hierarchy, with models freely flowing between VRAM, system RAM, and even solid-state drives. If this pans out, the constraint shifts from "how big is your GPU?" to "how big is your hard drive?"

The Insight MoE Unlocked

The Mixture of Experts architecture from earlier in this series already hinted at this. In a dense model, every token activates every parameter, so any weights sitting in slow system RAM become a bottleneck for every token. In an MoE model, each token only uses 8 out of 128 experts. If 6 of those 8 happen to be in VRAM and 2 are in RAM, you only pay the slow-memory penalty for those 2.

The logical next question: what if you could predict which 8 experts the next token will need and load them into VRAM before you need them?

Predictive Expert Prefetching

Several research teams are working on exactly this.

MoE-SpeQ (November 2025) uses a small, cheap helper model (called a "draft model") to predict which experts the next several tokens will route to. While the current token is computing through its 8 active experts, the system is already loading the experts that the next token will need from system RAM into VRAM. If the prediction is right (and it usually is - expert routing has patterns), the next token's experts are already waiting when it arrives. The I/O gets hidden behind computation.

But it's not the only approach.

DyMoE (2026) takes a different approach. It observed that expert selection between adjacent layers is partially predictable - if layer 20 routed a token to Expert 47, there's a higher-than-random chance that layer 21 will route it to a related expert. DyMoE exploits this to prefetch experts one layer ahead. The result: a 14.6x speedup over naive offloading. That's not an incremental improvement - that's a different category of performance.

DAOP (2025) flips the script entirely. Instead of moving expert weights to the GPU, it moves the token's activations (the intermediate calculations produced as a token passes through the model) to the CPU where the expert already lives. The key insight: "the activations transferred between CPU and GPU are approximately one ten-thousandth the size of expert weights." It's cheaper to send the work to the expert than to move the expert to the work.

Batching by Expert

There's another angle that's particularly elegant for prompt processing (the phase where the model ingests your entire input before generating a response). During prompt processing, you have many tokens to process at once. Different tokens route to different experts. What if you grouped the tokens by their expert routing?

Load Experts A, B, and C into VRAM. Process all the tokens that need those experts. Swap in Experts D, E, and F. Process their tokens. Minimize the number of expert loads by batching the work intelligently.

This is called token regrouping or expert-aware batching, and Speculative MoE (2025) demonstrated 1.7-2.4x speedup over baseline by pre-sorting tokens by routing path before they hit the expert layers.

The limitation: this mainly helps during prompt processing, where you have a batch of tokens to reorder. During generation, you're producing one token at a time - there's no batch to regroup. But for the initial prompt ingestion phase (which can be the slowest part of long-context interactions), it's significant.

What This Means Practically

The logical endpoint of this research: model size stops being limited by VRAM. Your GPU becomes a fast cache for whatever experts are actively needed. System RAM becomes the main storage. NVMe SSDs become the backing store for truly enormous models.

Imagine running a 200-billion-parameter MoE model on a consumer PC. The model itself might be 80 GB on disk. Your 12 GB GPU holds the currently-active experts and KV cache. Your 32 GB of system RAM holds the experts likely to be needed soon. The SSD holds everything else. Smart prefetching keeps the right experts flowing to the right tier.

You'd still need the compute to be fast when experts are in VRAM - this doesn't help with the actual matrix multiplications. But the memory constraint effectively disappears.

When Will It Land?

None of this is in consumer tooling yet. Ollama and llama.cpp do basic MoE inference and basic layer-level offloading (GPU layers vs. CPU layers, decided at startup, static for the session). They don't do predictive prefetching, expert-aware batching, or dynamic VRAM management.

The path from research to your machine typically looks like:

1. Research paper (done - 2025/2026)
2. Reference implementation in PyTorch (some exist)
3. Picked up by llama.cpp or vLLM, a popular LLM serving framework (not yet for most of these)
4. Trickles into Ollama and LM Studio (further out)

The project to watch is probably llama.cpp. It's historically been the fastest to adopt inference optimizations for consumer hardware - they already have basic expert offloading, quantized KV caches, and flash attention. Predictive expert prefetching is a natural next step.

The Series in Perspective

Let's zoom all the way out.

In 2020, running a frontier language model required a datacenter. Multiple high-end GPUs, hundreds of gigabytes of VRAM, cooling and power infrastructure. The model (GPT-3) had 175 billion parameters and would rank near the bottom of today's Arena leaderboard.

In 2026, a model with 26 billion parameters, 6.7x fewer than GPT-3, ranks ~#44 on that same leaderboard - outperforming GPT-4o. It runs on a single consumer GPU with 12 GB of VRAM.

The optimizations we covered in this series made that possible:

• FP16 halved everything (400 GB to 202 GB)
• GQA crushed the KV cache by sharing attention keys and values (202 GB to 112 GB)
• Sliding window bounded most layers to local context (112 GB to 60 GB)
• MoE made the model fast and offloading survivable (same memory, much less compute per token)
• Quantization compressed weights to 4-bit precision (60 GB to 19 GB)
• K=V equivalence halved the remaining global KV cache (19 GB to 18 GB)

None of these required a scientific breakthrough. Each was an engineering insight - often a simple one - applied at the right level of the stack. Use smaller numbers. Share some data structures. Don't look at what you don't need. Route to specialists. Round to fewer values. The genius isn't in any single trick; it's in stacking eight of them without the model falling apart.

And the next round of optimizations - the predictive prefetching, the smart memory hierarchies, the expert-aware batching - will push the boundary again. Not by making models smaller, but by making GPU memory less relevant.

The datacenter model on your desk was just the beginning.

---

Series start: How 400 GB Became 18

Written by Claude as dictated by Harald. Claude's memory requirements are left as an exercise for the reader.