# Local model memory worksheet

Version: 1.0
Updated: 2026-10-02
Method: https://suhailmohebi.com/insights/local-model-quantization-guide/

Purpose: work out whether a model, quantization and context fit your memory before you download, launch or blame the model.

Output: a filled worksheet with weights, KV cache per token, the total at your context and concurrency, and a chosen weight precision, KV precision and context.

## Use when

You are choosing a quantization or context length for local inference, or a set-up runs out of memory or slows sharply as context grows.

## Quick start

1. Take the weights figure from the model files on disk.
2. Compute KV bytes per token from the model’s `config.json`, then multiply by context and concurrent sequences.
3. Add measured overhead and headroom. If it does not fit, cut context or weight precision; test any KV precision change before trusting it.

Weights are the fixed bill; the KV cache is the meter that runs with context.

## 1. Weights

Best: the size of the weight files on disk, summed across shards. Count each copy of the weights once (some repositories ship a consolidated file beside shards); include a projector file only if you load it.

```bash
# Read-only: total size of the weight files in GiB (use *.gguf for GGUF files).
python3 -c 'import os, sys; print(f"{sum(map(os.path.getsize, sys.argv[1:])) / 2**30:.2f} GiB")' /models/my-model/*.safetensors
```

Otherwise estimate parameters × bits per weight ÷ 8, and expect more: real formats store scales too. In llama.cpp, Q4_0 packs 32 weights into 18 bytes (4.5 bits each) and Q8_0 into 34 bytes (8.5 bits); mixed formats such as Q4_K_M vary by tensor.

## 2. KV cache per token

For a standard full-attention transformer, including grouped-query attention:

KV bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per element

| Input | config.json field | Value |
|---|---|---|
| Layers | `num_hidden_layers` |  |
| KV heads | `num_key_value_heads`; if absent, `num_attention_heads` |  |
| Head dimension | `head_dim`; if absent, `hidden_size` ÷ `num_attention_heads` |  |
| Bytes per element | 2 for FP16 or BF16, 1 for FP8, 1.0625 for llama.cpp `q8_0` |  |

Multimodal configs often nest these fields under `text_config`. The formula over-counts or does not apply when layers differ:

- Sliding-window layers (`sliding_window`, or `sliding_attention` in `layer_types`) hold at most the window per sequence if the engine trims them.
- Hybrid models (`linear_attention` or `mamba` in `layer_types`, or `full_attention_interval`) pay per token only on attention layers, plus a fixed state per sequence.
- Latent attention (MLA, with fields such as `kv_lora_rank`) caches a compressed latent, not per-head K and V.

For these, count only the layers that grow with context and confirm against the engine’s own memory log.

```bash
# Read-only. Arguments: config.json, bytes per element, context tokens, concurrent sequences.
python3 - /models/my-model/config.json 2 32768 1 <<'PY'
import json, sys
c = json.load(open(sys.argv[1])); c = c.get("text_config") or c
bpe, ctx, seqs = float(sys.argv[2]), int(sys.argv[3]), int(sys.argv[4])
lt = c.get("layer_types")
L = sum(t in ("full_attention", "attention") for t in lt) if lt else c["num_hidden_layers"]
H = c.get("num_key_value_heads") or c["num_attention_heads"]
D = c.get("head_dim") or c["hidden_size"] // c["num_attention_heads"]
b = 2 * L * H * D * bpe
if c.get("kv_lora_rank"): H, D = "latent", c["kv_lora_rank"] + c["qk_rope_head_dim"]; b = L * D * bpe  # MLA
if not b: sys.exit("No attention layer found")
print(f"layers counted {L}, KV heads {H}, head dim {D}: {b:,.0f} bytes per token")
print(f"KV cache for {ctx:,} tokens x {seqs} sequence(s): {b * ctx * seqs / 2**30:.2f} GiB")
odd = [k for k in ("full_attention_interval", "kv_lora_rank") if c.get(k)]
odd += ["sliding_window"] if c.get("sliding_window") and c.get("use_sliding_window", True) else []
odd += ["mixed layer_types"] if lt and set(lt) != {"full_attention"} else []
if odd: print("CHECK:", ", ".join(odd), "- formula may not apply; confirm in the engine log")
PY
```

## 3. Total and budget

| Line | GiB | Source |
|---|---|---|
| A. Weights |  | file sizes |
| B. KV cache = KV per token × context × concurrent sequences, plus any fixed sliding-window or recurrent state per sequence |  | §2 |
| C. Draft model weights and cache, if used |  | file size and §2 |
| D. Runtime overhead, activations, compute buffers |  | measured at peak: engine log or monitor; before the first launch, decide on headroom alone |
| E. Headroom: OS, other apps, margin |  | your choice |
| Total = A + B + C + D + E |  |  |
| Budget: usable GPU memory, or the Mac GPU limit in §4 |  | driver or OS report |

- Use one unit throughout. Mixing GB and GiB alone is a 7.4% error.
- Set the context explicitly: some servers default to the model’s full trained length.
- Check how your server counts concurrency: some share one cache pool across sequences, so the configured context is the total.

## 4. Unified-memory Macs

By default the GPU may use only part of unified memory, so budget against that limit, not total RAM. A conservative starting heuristic, not an official limit: keep weights near or below half of unified memory.

```bash
# macOS, read-only: override in MB (0 = none), then the default limit in bytes (needs MLX;
# llama.cpp logs it as recommendedMaxWorkingSetSize).
sysctl iogpu.wired_limit_mb
python3 -c 'import mlx.core as mx; print(mx.device_info()["max_recommended_working_set_size"])'
```

MLX’s documentation describes raising the limit on macOS 15 or later with `sudo sysctl iogpu.wired_limit_mb=N`, N above the model size in MB and below total memory. That changes a system setting for every app: do it deliberately, record it with your results, and never treat it as a default step.

## 5. Decision procedure

1. Weights first: the largest weight precision that still leaves room for the context the job needs.
2. KV precision second, as its own decision. An 8-bit cache is the usual first saving where the engine supports it; test it on your task at target depth first.
3. Do not buy context with a 4-bit cache. SGLang’s own evaluation shows one roughly halving a hard reasoning benchmark on a 120B model while an easy one held, and notes that degradation may be worse at long context and on smaller models.
4. Still short? Pay honestly: drop the draft model or unused parts such as a vision encoder, change the weight build, or cut concurrency or context.
5. Test coherence with reasoning at the target depth, not needle recall alone.

## Worked example

Hypothetical model: 48 layers, 8 KV heads, head dimension 128, FP16 cache, one sequence, no draft model, 32 GiB of usable GPU memory. The job needs 32,768 tokens; 65,536 would help.

```text
KV per token = 2 x 48 x 8 x 128 x 2 bytes = 196,608 bytes (192 KiB)
KV at 32,768 tokens = 196,608 x 32,768 = 6,442,450,944 bytes = 6.00 GiB
KV at 65,536 tokens = 12.00 GiB with FP16, 6.00 GiB with an FP8 cache (6.38 with q8_0)
Fixed lines: weights 18.6 + overhead 1.5 (measured) + headroom 2.0 = 22.1 GiB
32,768 at FP16:  22.1 + 6.0  = 28.1 GiB -> fits
65,536 at FP16:  22.1 + 12.0 = 34.1 GiB -> does not fit
65,536 at FP8:   22.1 + 6.0  = 28.1 GiB -> fits, quality untested
```

Decision: run 32,768 tokens on the FP16 cache now. Trial 65,536 on an FP8 cache only after it passes the job’s reasoning checks at that depth.

## Stop if

- The total with headroom exceeds the budget: cut context or precision rather than launching and hoping.
- The config shows sliding-window, hybrid or latent attention and the engine log has not confirmed the per-token cost.
- Measured memory exceeds the worksheet by more than the headroom: find the missing line before going deeper.
- A cache-precision change is about to ship without a reasoning test at target depth.
- Raising a system memory limit is proposed as a default step.

## Sources

- Hugging Face: LlamaConfig parameters — https://huggingface.co/docs/transformers/model_doc/llama (checked 2026-10-02)
- ggml-org/llama.cpp: ggml-common.h block layouts — https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-common.h (checked 2026-10-02)
- ml-explore/mlx-lm: README, Large Models — https://github.com/ml-explore/mlx-lm (checked 2026-10-02)
- vLLM: The State of FP8 KV-Cache and Attention Quantization in vLLM — https://vllm.ai/blog/2026-04-22-fp8-kvcache (checked 2026-10-02)
- SGLang: Quantized KV Cache — https://docs.sglang.io/docs/advanced_features/quantized_kv_cache (checked 2026-10-02)