Will this model fit? A local LLM memory calculator

§1The calculator

§2The formula

Memory needed is four parts added together. Everything is counted for one request at a time; a server holding several conversations pays the KV cache once per conversation.

A worked example, for Qwen 3.8 27B with a 32,768-token context and a 16-bit cache. Its cache is small for its size, because only 16 of its 64 layers are full attention. Each part is rounded to 0.1 GiB:

Then the verdict compares the total with a budget. On a discrete GPU the budget is its memory; what does not fit can spill into system RAM, minus 8 GiB kept for the operating system. On a Mac it is macOS’s GPU limit. On a DGX Spark it is the unified memory minus 8 GiB.

§3Every assumption

• Bits per weight are effective rates, not format names. The GGUF rates are file size ÷ parameters on llama.cpp’s own reference model: Q2_K 3.17, Q3_K_M 4.0, Q4_K_M 4.9, Q5_K_M 5.7, Q6_K 6.57 and Q8_0 8.52. They sit above the nominal rates because each mix keeps some tensors at higher precision. NVFP4 is 4.5 (4-bit values with one FP8 scale per 16), FP8 is 8 and BF16 is 16.
• Mixed-precision checkpoints run larger. Exports that keep attention, embeddings or shared experts in BF16 are bigger than the rate suggests. Where a published file size is known for the chosen model and quantization, the calculator uses that file for the weights and the verdict, and shows the formula’s figure in a note.
• Cache precision. FP16 and BF16 are 2 bytes per element and FP8 is 1. llama.cpp’s q8_0 and q4_0 store 32 values with one 16-bit scale, so 8.5 and 4.5 bits. The choices follow the weights: GGUF files run in llama.cpp, which offers FP16, q8_0 and q4_0 caches; NVFP4, FP8 and BF16 checkpoints run in vLLM or SGLang, which offer FP16/BF16 and FP8.
• Overhead is a flat allowance. 1.5 GiB plus 5% of the weights covers the CUDA or Metal context, compute buffers and activations for one request. Engines that reserve a fixed share of the card up front, such as vLLM, will show more memory in use than this; that extra is mostly spare cache, not a requirement.
• Macs. By default macOS lets the GPU wire about 75% of unified memory on the sizes listed here. That figure is documented by the community, not by Apple. sudo sysctl iogpu.wired_limit_mb=<MB> raises it until the next reboot, and 0 restores the default. The calculator says when a setup fits only after raising it, and keeps 8 GiB for macOS.
• Offload. “Fits with offload” means the total fits in GPU memory plus free system RAM. It will run, but every token waits on the slower memory. Mixture-of-experts models suffer less, because each token touches only a few experts.
• Custom models are treated as plain transformers: every layer is full attention with separate keys and values. A hybrid or sliding-window model entered that way will show too large a cache.

§4The cases a simple formula gets wrong

• Mixture of experts. “3B active” or “6B active” describes compute per token, not storage. The weights line counts every expert, so gpt-oss-120b is sized as 117B parameters even though about 5.1B work on each token.
• Hybrid attention. Qwen 3.8 27B, Qwen 3.6 35B-A3B and Flash-Next use full attention on only one layer in four. The other layers keep a fixed recurrent state that does not grow with context. That is why the 27B pays 64 KiB per token at 16 bits, 32 KiB at FP8, and why its full 262K window costs exactly 8 GiB of FP8 cache.
• Sliding windows. Half of gpt-oss’s layers attend to the last 128 tokens and five of every six of Gemma 4’s to the last 1,024. Those layers stop growing at the window. Gemma 4’s global layers share one tensor for keys and values, so they are counted once; an engine that stores both would need more for that part.
• Flash-Next’s n-gram table. About 51B of its 180B parameters are an n-gram embedding table that is looked up, not computed through. It can live in host RAM or be read from SSD or NVMe, which is how every small Flash-Next setup works. The calculator splits it out and says when a setup fits only that way, rather than pretending it must sit in GPU memory. GGUF builds keep the table at 4-bit or above and NVIDIA’s NVFP4 export stores it in FP8, so it keeps that precision whichever weight format you pick. Its source-checked sizes and runtimes are in the Flash-Next requirements.
• Not modelled. Vision encoders’ image buffers, speculative drafters (a second model needs its own weights and cache), Flash-Next’s sparse-attention indexer cache, and several concurrent requests. A drafter is often what decides whether a long window fits; SGLang vs vLLM shows that trade on one card.

§5Estimates against measurements

Two of my own measurements check the arithmetic. On the RTX 5090, Qwen 3.8 27B in NVFP4 with an FP8 cache held a physical KV pool of 114,688 tokens. The formula gives exactly 3.5 GiB for that pool, and the engine’s startup log reported the same 3.5 GiB. On the 128 GB M4 Max, the MLX 8-bit weights are about 29.5 GB; Q8_0’s rate gives 29.6 GB.

The calculator is weakest where checkpoints mix precisions. For Flash-Next in NVFP4 it estimates 115 GiB of weights against NVIDIA’s published 123.57 GiB, because that export keeps attention and shared experts in BF16. So where a published file size is known, the calculator uses it. And when the headroom is smaller than the estimate’s typical error (10% of every formula-sized part, 3% of a published file), the verdict says “measure first”.

How these figures were gathered. Layer counts, attention layouts, KV heads, head dimensions and parameter counts were read from each model’s Hugging Face configuration, parameter count and model card on 8 October 2026; the source of every value is recorded in the calculator’s code. The quantization rates come from llama.cpp’s quantize tool and NVIDIA’s NVFP4 description, read the same day. The two cross-checks are my own runs from August 2026.

Primary sources

• Qwen 3.8 27B on Hugging Face and Qwen 3.8 Flash-Next
• Qwen 3.6 35B-A3B, gpt-oss-120b and Gemma 4 31B
• llama.cpp: the quantize tool’s size table and the cache block formats
• NVIDIA: introducing NVFP4