How I Choose a Quantization for Local Inference

Get quantization right and a 27B model runs comfortably on a single consumer card. Get it wrong and you either run out of memory, or run a model that is fluent and subtly wrong. Both failures look like something else until you measure them.

§1What quantization actually changes

A model is a set of numbers. Each weight is stored as a fixed number of bits. More bits means the stored numbers sit closer to the model's original precision, which usually means better quality — at the cost of memory.

Quantization reduces those bits. It does not delete the model; it rounds the numbers. A 4-bit weight stores one of sixteen values; an 8-bit weight stores one of 256. The 8-bit version can express finer distinctions, so it usually answers more precisely.

§2The formats that actually matter

You do not need to know every format. You need the handful that show up in practice:

The Q4 family has sub-formats (Q4_0, Q4_1, Q4_K_M). The _K_M variant is a mix that keeps the important weights more precise. It is the default for a reason. Do not chase the exotic variants; start with Q4_K_M and Q8, and only go lower if memory forces you.

§3The memory math that decides everything

VRAM is shared. The model weights are the fixed bill. The KV cache is the growing bill. Everything else — runtime workspaces, CUDA graphs, a speculative drafter when you use one — competes for what is left.

So the real question is never "how big is the model?" It is "how much memory is left for context after the model loads?" On a 32 GB card, roughly:

This is the whole lesson in one direction: lower quantization frees memory for context, and context is the thing you are actually buying. A model that fits with room for 64K context is more useful than a model that fills the card at 8-bit and can only hold 8K.

§4The quality trade-off, honestly

Here is what I have observed, and what I would tell someone who asks me directly:

• BF16 → FP8: usually no measurable difference on real tasks. FP8 is a free win when your card supports it natively.
• FP8 → Q8: rarely noticeable on chat, coding, or tool use. Sometimes visible on long recall or structured output.
• Q8 → Q6: the first place quality starts to drop, but gently.
• Q6 → Q4: noticeable for some people, on some tasks. Recall and tool calling are the first things to slip.
• Q4 → NVFP4: the point where you are betting that "good enough" is still good enough.

The pattern is a cliff after the cliff. The first few steps lose almost nothing. The last step can lose a lot — and you usually only find out by testing, not by reading.

§5How to test a quantization instead of guessing

Do not trust the label. Test the model the way you will actually use it. My field-note contract records nine things for every configuration, and a quantization is no different:

1. Model identity + quantization — exact filename and bits.
2. Configured context — what the runtime was told to reserve.
3. Task and input shape — the real job, not a synthetic prompt.
4. Recall — can it answer a question it was asked to remember?
5. Tool calling — does structured output still work?
6. Instruction following — does it obey the format you asked for?
7. Latency and memory — tokens per second, VRAM after the run.
8. Observed failure — what broke, and how.
9. Cloud comparison — does it match a cloud model on the same task?

A quantization that returns fluent text but breaks tool calls is not faster. It is broken. If a configuration passes all nine, it is good enough. If it fails one, the question is not "is 4-bit bad?" It is "does this task need the thing that broke?"

§6The KV cache is the hidden cost

Here is the part most guides skip. A lower quantization frees memory for context, but a speculative drafter — the thing that makes local inference fast — also uses that memory. So the cheapest model is not automatically the fastest, because the fast recipe may not fit in the memory you just saved.

I measured this directly when I built the RTX 5090 serving recipes: the fastest profile was not the longest-context one. D-Flash 2 stayed my validated 32K speed profile. DSpark, which used less VRAM, could reach 112K. Quantization, context, and speed are three lanes that share one tank. You cannot optimize all three at once.

§7What I actually run, and why

I do not run one quantization for everything. I run the one that matches the machine and the job:

• On the RTX 5090: NVFP4 weights with an FP8 KV cache — the 4-bit format the card accelerates natively, adopted only after it passed recall, tool-calling, and stability gates, with named profiles for fast 32K chat and longer-context agent work.
• On the Mac: Q4_K_M or 8-bit, chosen by how much unified memory the job can spare — the same RAM-based choice I documented for Qwen 3.8.
• Anything that must be right: the highest precision I can afford, tested before I trust it.

The rule is not "always use the smallest." The rule is use the largest quantization that still leaves room for the context this task needs. A coding task that needs 64K context should give up weight precision for cache room, because the context is the bottleneck, not the model.

§8The mistakes I made

Mistake 1: chasing the smallest number. Saving a few gigabytes on weights bought nothing when the thing I actually wanted — a speculative drafter plus long context — no longer fit beside them in the same recipe. Smaller was not better. It was just smaller.

Mistake 2: trusting the advertised context. A model can advertise 128K while the loaded configuration has far less room. The effective limit is the smaller of the declared length and the physical KV pool. Always measure what fits, not what is printed.

Mistake 3: comparing quantizations on one synthetic prompt. A high tokens-per-second on a short prompt means nothing. Test on the real task, at the real context length, and repeat it. One run per configuration is not a measurement.

§9The questions people actually ask