A 32 GB Mac runs a 27B model at 4-bit with modest context. A 96–128 GB Mac runs the 8-bit conversion with real headroom. Speculative decoding is the speed lever — on my M4 Max a DFlash 2 drafter took decode from 16.7 to 47.2 tokens per second. The KV cache, not the weights, is where your memory actually goes. And when prefill dominates the job, a GPU box is a different class of machine: 46× faster at prefill on the same model family. Everything below is the measured detail.
§1What a Mac buys you
Unified memory is the quiet advantage. A 128 GB Mac holds model weights, the growing context cache, and the surrounding tools in one pool that no consumer GPU matches for size. That is what lets a laptop run a model that sits in the same performance conversation as frontier cloud models on selected published benchmarks — with real wins and real losses, honestly read.
The second advantage is the boundary. A local model reads private drafts, code, and documents without any of it leaving the machine — the practical case for local AI is privacy and control first, capability second. The trade is that every gigabyte is yours to budget, which is what the rest of this guide is about.
§2Pick the quantization by RAM, not ambition
My conservative starting rule on Apple silicon: keep model weights near or below half of unified memory — macOS, the KV cache, activations, and every open app need the rest. For a 27B-class model, the current MLX weight files are approximately 16.1 GB at 4-bit, 29.5 GB at 8-bit, and 54.7 GB at BF16. Those are starting costs, not the whole runtime bill.
| Unified memory | Sensible choice | Why |
|---|---|---|
| 16 GB | A smaller model | The 27B 4-bit weights alone are ~16.1 GB; it is not a sensible fit. |
| 24 GB | Smaller is safer | 27B 4-bit is a tight experiment, not a daily default. |
| 32 GB | 27B at 4-bit | Modest context, memory-heavy apps closed, watch pressure during real tasks. |
| 48–64 GB | 4-bit first | Trial 8-bit at 64 GB only if measured quality gains justify the smaller cushion. |
| 96–128 GB | 27B at 8-bit | ~29.5 GB of weights leaves real room for context, cache, and tools. |
The full reasoning, the exact receipt for my own 128 GB M4 Max, and the honest cloud comparison live in the Qwen 3.8 dive; the general theory of what each format trades away is in the quantization guide. The Qwen dive's download rule: if you are unsure, choose the smaller quantization — a model that finishes reliably beats a larger one that swaps, stalls, or crashes. The quantization guide sharpens the same trade from the other side: use the largest quantization that still leaves room for the context the task needs.
§3Make it fast: speculation is the real lever
Raw decode on a big Mac is serviceable, not thrilling. The measured lever is speculative decoding, where a small drafter proposes tokens and the big model verifies them. Measured on my M4 Max, on two mutually exclusive builds of the model, each against its own baseline:
| Build (mutually exclusive) | Measured decode |
|---|---|
| Native multi-token prediction (MTP) build | 18.5 → 22.9 tokens per second, about +24% |
| DFlash 2 drafter build | 16.7 → up to 47.2 tokens per second, up to +183% |
Two cautions from the same dive. First, MTP and a DFlash drafter are alternatives, not a stack. Second — the one that nearly reverted a working setup — judge speculation by decode rate, not by the logged end-to-end tokens per second: the logged figure includes prefill, and comparing it against a decode baseline makes a working drafter look broken.
§4Context is a budget, and the KV cache is the bill
Weights are the fixed cost; the KV cache is the growing one. It has its own precision knob, and that knob does not behave like the weight knob: in my RTX 5090 serving work an FP8 KV cache was effectively free, while the engine's published table — measured on a larger reasoning model, not my build — shows a 4-bit KV cache halving a reasoning benchmark. Trust the direction, verify the magnitude on your own setup: choose weight precision first, and treat KV precision as a separate decision with its own test.
Two more context rules that survived measurement. The advertised window is not the effective one — the real limit is the smaller of the declared length and the memory actually left after the weights load. And a long window is only worth paying for if behavior holds at depth: a 40-check graded suite at four depths of a 262k window — run on the GPU workstation with a different model, one cold pass per depth and labeled as such — found tool calls clean everywhere and the test harness failing before the model did. The lesson transfers to any long window: grade the run, not the vibe.
§5When the Mac is the wrong tool
I run the same 27B family on both a 128 GB M4 Max and a 32 GB RTX 5090. The gap is phase-dependent, and that is the whole routing lesson:
| Phase | RTX 5090 advantage | What it means |
|---|---|---|
| Prefill (reading the prompt) | ~46× | Long-document and big-context jobs belong on the GPU. |
| Decode (writing the answer) | ~5× | Meaningful, but not the same class of gap. |
| Decode with speculation | ~3× | A drafted Mac closes much of the writing gap. |
| Cached follow-up turns | ~1× | Conversation on warm context is nearly even. |
Route by phase, not loyalty: interactive, private, conversation-shaped work suits the Mac; prefill-heavy batch work suits the GPU box. The 5090 side of that story — servers, recipes, and remote access — is its own dive, the RTX 5090 workstation build.
§6The operating discipline that makes any of this true
Every number above was measured under a recorded configuration, with its limits recorded next to it — the long-context suite is single cold passes per depth and says so; the KV cliff is a published table, not my rerun. The working checklist is the quantization guide's nine-point test — model identity and quantization, configured context, the real task, recall, tool calling, instruction following, latency and memory, observed failure, and a cloud comparison — logged as dated field notes. A configuration that returns fluent text but breaks tool calls is not faster; it is broken.
One run per configuration is not a measurement. Repeat the run, at the real context length, on the real task, before trusting any tokens-per-second figure — including the ones in this guide, on your machine.
§7The questions people actually ask
Which Mac do I need to run a local LLM well?
A 32 GB Mac runs a 27B model at 4-bit with modest context. 96–128 GB runs the 8-bit conversion with real headroom for context and tools. On 16 GB, choose a smaller model — the 27B 4-bit weights alone are about 16.1 GB.
How fast is local inference on a Mac?
Measured on a 128 GB M4 Max with a 27B-class model: a DFlash 2 speculative drafter took decode from 16.7 to 47.2 tokens per second at best. A separate native-MTP build measured 18.5 → 22.9, about +24%.
Is a local model as good as a cloud model?
On selected published benchmarks a 27B open model now sits in the same performance conversation as a frontier cloud model, with real wins and real losses. Same class on some tests is not the same model on all work — test on your actual task.
Do I need speculative decoding?
It is the biggest single speed lever on a Mac: about +24% from a native-MTP build, up to +183% from a DFlash 2 drafter build, measured on the same machine. Judge it by decode rate, not the end-to-end logged figure, which includes prefill.
When should I use a GPU box instead?
When prefill dominates. On the same model family the RTX 5090 measured ~46× faster at prefill but only ~3× with speculation and ~1× on cached turns. Route by phase of work, not by machine loyalty.
The decision path in one view
- Pick the quantization by unified memory: 4-bit at 32 GB, 8-bit at 96 GB and up, smaller models below that.
- Add speculation before adding hardware — it is the measured 2–3× lever on a Mac.
- Budget the KV cache separately from the weights; FP8 KV is the evidence-backed default, 4-bit KV is a published cliff.
- Route by phase: conversation and private work on the Mac, prefill-heavy jobs on the GPU box.
- Trust only configurations that passed the nine-point test on the real task — and repeat the run before trusting a speed number.