The thousand-token window

We're at 112 GB. The KV cache is down to 58 GB thanks to GQA, but it's still the biggest single cost. Every layer in the model is storing attention notes for every token in the 256,000-token context. All 30 layers, all 256k tokens.

Sliding window attention asks: do they all need to?

Reading a Book

Think about how you read. To understand the sentence you're on right now, you mostly need the surrounding paragraph. Maybe the last few paragraphs. You don't need to re-read the first page of this blog post to parse the current sentence.

That's the intuition behind sliding window attention. Instead of every layer looking at the entire context, most layers only look at a small neighborhood - the last 1,024 tokens. A "sliding window" that moves forward as the conversation progresses.

But you do occasionally need the big picture. If I reference "the 400 GB model from the intro," you need to connect that to something from much earlier in the series. Some layers need global reach.

The solution: interleave. Most layers get the cheap sliding window. A few layers get expensive full-context global attention. In Gemma 4, out of 30 layers, 25 use a 1,024-token sliding window and only 5 use full global attention, spread evenly through the stack - so no sliding window layer is ever more than a few steps away from a global layer that can feed it the big picture.

The Math

This changes the KV cache calculation completely. Sliding window layers have a fixed cache size regardless of context length. Whether your conversation is 10k tokens or 256k tokens, each sliding window layer only ever stores 1,024 tokens worth of Keys and Values.

Before sliding window, all 30 layers cached 256k tokens:

30 layers x 256,000 tokens = 7,680,000 layer-tokens to cache

After:

25 layers x   1,024 tokens =     25,600 layer-tokens (fixed)
 5 layers x 256,000 tokens =  1,280,000 layer-tokens (scales)
                              ---------
                              1,305,600 layer-tokens

That's roughly 17% of what we had. The KV cache drops from 58 GB to about 5.5 GB.

But How Does Information Flow?

This is the natural objection. If layer 20 can only see the last 1,024 tokens, how does it know about something from 50,000 tokens ago?

The answer is that information propagates upward through the stack. A global attention layer at, say, layer 15 sees the entire context. It processes that information and encodes it into its output. That output feeds into layers 16, 17, 18 - all sliding window layers - but they're now working with representations that already contain long-range information. The global layer baked it in.

Think of it like a newsroom. Most reporters (sliding window layers) focus on their local beat - the last few hours of events. But a few editors (global layers) read everything and write summaries that get distributed to the whole team. The reporters don't need to read the entire archive themselves; they work with the editors' summaries.

It turns out models are surprisingly resilient to having very few global layers. The information from early global layers propagates so effectively through the stack that even a sliding window layer near the top indirectly "sees" the full context through what the global layers already processed and passed along.

Who Decides Which Layers Get What?

The engineers designing the model choose the pattern before training - which layers are global, which are sliding window, what the window size is. Then the model learns to work within those constraints during training. The sliding window layers learn to extract local patterns (syntax, nearby references, grammar), and the global layers learn to handle long-range connections (callbacks, overall coherence, topic tracking).

How do they pick the right pattern? Mostly experimentation. Try different configurations, train at smaller scale, benchmark quality, iterate. There's an intuition that later layers (closer to the output) benefit more from global context, and that the final layer should always be global. But the specifics are empirical.

Sidebar: Graduate student descent

The Running Total

Step	Weights	KV Cache	Overhead	Total
Naive (FP32)	104 GB	294 GB	2.6 GB	~400 GB
+ FP16	52 GB	147 GB	2.6 GB	~202 GB
+ GQA	52 GB	~58 GB	2.6 GB	~112 GB
+ Sliding window	52 GB	~5.5 GB	2.6 GB	~60 GB

The KV cache went from the dominant cost (294 GB in the naive model) to under 6 GB. Three optimizations - GQA, sliding window, and FP16 on the cache - turned the context memory from an impossible problem into almost a rounding error.

The problem has completely flipped. It's now all about the 52 GB of model weights. Which means we need a fundamentally different kind of trick.

Next up: a model with 128 brains, only 8 of which are awake at any given moment.

---

Next up: The Hospital With 128 Doctors: Mixture of Experts | Series start: How 400 GB Became 18

Written by Claude as dictated by Harald, whose attention definitely never wandered while writing about attention windows.