Conceptual GPU board with fixed dark model blocks and an expanding green KV cache
The weights are the fixed bill. The cache is the growing one — and it has its own precision knob.

§1The cache is the context

The KV cache is the model’s working memory of everything already in the window. Every token you feed it deposits keys and values that every later token’s attention will read. Weights are a fixed bill paid once at load; the cache is a meter that runs with context length. When a card “can’t hold long context,” it is almost never the weights that failed to fit — it is this meter.

The 27B I serve makes the arithmetic unusually visible, because it is a hybrid: only 16 layers are full attention (grouped-query, 4 KV heads, head dimension 256), and those are the only layers that pay per token — 32.8 KB per token at FP8, 65.5 KB at BF16. The remaining layers keep a fixed-size recurrent state (78–154 MB per slot depending on dtype) that never grows with context. That hybrid design is the only reason a 262k window is even conceivable on a 32 GB card: a conventional all-attention 27B would pay per token on every layer.

§2The budget, in real numbers

Numbers from my live serving config, all of which cross-check against what I already published:

THE BUDGET RULE

On a fixed card, weights, cache, and drafter share one pot. Every choice is a context choice — including the ones that don’t mention context.

§3FP8: the free half of the knob

Dropping the KV cache from BF16 to FP8 doubles capacity, and the best available evidence says it costs approximately nothing. A four-model study by the vLLM project concluded FP8 is the right default: twice the context, negligible accuracy loss, and no speed penalty — because FP8 KV quantizes the attention computation itself, whereas the storage-only compression schemes it compared keep the math in BF16 and pay a dequantization toll on every read (all of them measured strictly slower).

My own receipts agree as far as they go: every matched run in dive 35 was served on an FP8 KV cache and passed the recall, tool-calling, and vision gates — including three-needle recall at 178,014 actual tokens. FP8 KV is not a compromise I tolerate; it is the setting the rest of the stack is built on.

§4FP4: the cliff, in the engine’s own table

Halving the cache again looks irresistible on paper — a 4-bit KV cache would put the full 262k window back within reach of this card. The engine’s own documentation is why I have not done it. SGLang’s published evaluation of 4-bit KV on a 120B reasoning model shows AIME-25 falling from 0.753 to 0.353 and GPQA-diamond from 0.508 to 0.320 — half the reasoning score, from a knob that never touched the weights. The same source notes smaller models tolerate it worse, and that the error grows with context length.

Read that last part again, because it is the trap in full: the knob that promises more context is the knob that degrades fastest with context. The 4-bit cache buys you a longer window and then corrupts precisely the long-window workloads you bought it for. Milder storage-only 4-bit schemes exist — one buys 3.4× capacity for one to four accuracy points — but they carry a 10–60% latency toll, and the one I would want is not currently available in my engine anyway.

To be exact about provenance: I did not reproduce this cliff myself. The numbers are the engine’s published table, measured on a different, larger model than mine. I trust the direction, not the exact magnitude, and I declined to volunteer my stack for the experiment. Sometimes the receipt you act on is someone else’s — as long as you say so.

§5Why the cache is more fragile than the weights

The asymmetry has a clean mechanical explanation:

The hybrid architecture adds one more precision knob most guides never mention: the recurrent-state dtype on the non-attention layers. The serving cookbook flags it as an explicit accuracy gate to validate per workload — and I have to report that my own 48-prompt config sweep never tested it, because every config in that sweep ran the same state dtype. Forty-eight byte-identical outputs proved my prefill knobs were output-preserving; they proved nothing about state precision. That test is still owed. It is not even a guaranteed speed win: in my engine’s published numbers the higher-precision state won one speculative combination and lost another.

§6The decision procedure

1. WEIGHTS FIRST   pick weight precision by the memory math
                   (deep dive 36) — it sets the whole budget
2. KV = FP8        the default with evidence behind it;
                   2× context, ~free, full speed
3. NEVER KV4       do not chase context with a 4-bit cache —
                   it breaks the workloads it enables
4. WANT 262K?      pay honestly: drop the drafter, or drop
                   the vision tower, or change the weight build
5. TEST COHERENCE  probe with reasoning at target depth, not
                   needles — retrieval survives after
                   coherence is already gone

Point 5 matters most. A needle test asks the cache “do you still have the fact?” A reasoning test asks “can you still think across what you have?” The published cliff shows reasoning collapsing while the model still answers — so a needle pass at a big context is not evidence the cache precision is safe. Grade the property you actually need, the same lesson dive 38 applied to rules at depth.

The exact local receipt

KV PRECISION RECEIPT · CONFIG OF RECORD · 26 AUG 2026

Model
Qwen 3.8 27B · NVFP4 weights · hybrid attention
Attention layers
16 full-attention · GQA 4 · head dim 256
KV per token
32.8 KB at FP8 · 65.5 KB at BF16
KV cache
FP8 · 3.59 GiB pool · 114,688 tokens
Recurrent state
Fixed-size slot per sequence · dtype gate still untested
Not adopted
4-bit KV (published reasoning cliff) · storage-only 4-bit (latency toll, unavailable)
Sources
Own matched runs (dive 35) · engine-published KV4 table · vLLM four-model KV study
Decision
NVFP4 weights + FP8 KV stands; 262k waits until it can be paid for honestly

The compact rule

  • The cache is the context: weights are the fixed bill, KV is the meter.
  • Hybrid attention is why long context is affordable here — only 16 of the layers pay per token.
  • FP8 KV is the evidence-backed default: 2× capacity, ~zero loss, full speed.
  • 4-bit KV is a coherence cliff, and it degrades fastest exactly where it promises most — at depth.
  • Cache errors compound through attention; weight errors do not. That is the whole asymmetry.
  • Acting on someone else’s receipt is fine — claiming it as yours is not. The cliff numbers are the engine’s, disclosed as such.
  • Test KV precision with reasoning at depth, never with needles alone.

Primary sources