Speculative decoding, measured on my M4 Max

§1The five-second version

A language model normally writes one token per forward pass, and on a 27B model every pass is expensive. Speculative decoding puts a much smaller drafter in front: it guesses several tokens cheaply, and the big model checks the whole guess in a single pass. Accepted guesses are nearly free speed. Rejected guesses cost only the check. Under greedy decoding the output is identical to what the big model would have written alone—the drafter proposes, the target disposes.

§2Two ways to get a drafter

There are two flavors, and on my runner they are mutually exclusive per model—the model’s own anatomy decides which one you get.

• Native multi-token prediction (MTP). Some builds ship extra prediction weights alongside the main model. The drafter is part of the checkpoint; you switch it on rather than install it.
• An external drafter. DFlash 2 is a separate ~2B Apache-2.0 model trained specifically to track Qwen3.8-27B’s outputs. It bolts onto any compatible target that has no draft head of its own.

I measured both, because I run two different Qwen3.8-27B community derivatives on this Mac: a vision-capable 4-bit build (17 GB, quantization group 128) that ships its own MTP weights, and an abliterated 4-bit build (14 GB, group 64) with no draft head, which gets DFlash 2. What abliteration is—and why I use it at all—is covered in deep dive 05; this note treats that build purely as an inference target.

§3The bench rules

Every number below was collected the same way, changing one variable at a time:

PROMPT        same shape every run · ~2,150 tokens
STREAMING     on · first-token time (TTFT) recorded
DECODE RATE   completion tokens ÷ (wall − TTFT)
RUNS          three per configuration · mean reported
CONCURRENCY   1 request · one config change per A/B
CONTEXT       128K configured · 262K native

The decode-rate definition is not pedantry. It is the entire difference between §4 and §5 of this note.

§4The numbers

Each mean is three runs, not one. The +131% row is [35.8, 40.6, 39.7] tok/s against a [16.9, 16.0, 17.2] baseline; the MTP row is [23.8, 25.7, 19.4] against [18.5, 17.9, 19.1]. Cold first-token time on the drafted build also fell on its own, 15.41 s to 11.41 s, before any cache was involved. Separately, a warm prefix cache moved time-to-first-token on the long prompt from 12.34 s to 1.23 s—a 90% cut before the first output token exists.

Three readings from that table:

• The fastest lane is the reasoning lane. Thinking tokens are highly predictable, so the drafter lands more of them; and that lane runs neutral penalties, which keeps the verification path free of logits processors. Predictable text accelerates best.
• Greedy stabilises acceptance rather than raising the ceiling. The card’s greedy settings narrowed acceptance to 65–72% and landed a slightly lower decode rate than the sampled lanes.
• A finetuned target costs acceptance. DFlash 2 publishes 2.7–3.4× at batch 1 against stock Qwen3.8-27B weights on datacenter GPUs. Against an abliterated derivative on this Mac it delivered 2.1–2.8×—the drafter was trained to track stock outputs, and every behavioural deviation you finetune in is a guess it has never seen.

The failed first pairing

The table above was not the first attempt of the day. Earlier that afternoon I bolted DFlash 2 onto the MTP build—the target that already ships its own draft weights—under a stricter harness: temperature 0, fixed seed, 128-token outputs, exact SHA-256 output-parity checks, on a machine deliberately left in its real multi-service state.

A quantized-draft variant did better—+1.5% at 4K, +26.1% at 16K—but was slower end to end at both contexts, and the median decode gain sat under the 15% adoption gate I had set in advance. So that configuration shipped disabled, with exact output parity confirmed and the draft files retained. Nothing stopped me bolting an external drafter onto a build that already carries draft weights; the runner allowed it, and the numbers vetoed it. Re-pairing the drafter with the draft-headless build the same evening is what unlocked every gain in the table above. Acceptance barely distinguishes the two experiments—roughly 58–62% on the wrong target, 65–82% on the right one. What changed the outcome was everything around the drafts: the evening run used the quantized draft, generations far longer than the afternoon’s 128-token outputs, and lanes whose text the drafter predicts well. Acceptance alone never tells you the speedup.

§5The number that nearly reverted a working config

My runner logs speculative-decoding throughput like this:

generation complete: 181 tokens, 11.3 tok/s, acceptance=74.0%,
  phases[prefill=10970.8ms draft=232.9ms verify=244.8ms
         replay=24.1ms commit=49.2ms]

11.3 tok/s. My no-drafter baseline decoded at 16.7. For a few minutes on 20 August, the working conclusion was “DFlash 2 is a regression—revert it.”

The logged figure is end-to-end: total tokens divided by total wall time, prefill included. With a ~2,150-token prompt, prefill alone was ~11 of the 16 seconds. The same run measured as a decode rate—completion tokens over the window after the first token—was 35.8 tok/s. On this target, that is the entire difference between “broken” and “a 2.3× speedup.” One phase dominated the denominator, and the log never said so.

• Never compare an end-to-end figure against a decode baseline. Both are legitimate; crossing them manufactures a regression or a miracle.
• Warm-cache runs get the opposite trap. With TTFT collapsed to ~1 s, cache-restore work lands inside the decode window and decode looks slower while the run is clearly faster. Compare warm runs end-to-end only.

§6Read the derivative’s card, not the base card

The second config error of the session: I applied the base Qwen sampling recipe—temperature 1.0, top-p 0.95, top-k 20—to the abliterated build, because that is what the original model documents. The derivative’s own model card explicitly rejects those values: it wants greedy temperature 0, a repetition penalty of 1.15 it calls essential (without it, greedy decoding loops on imports and boilerplate), no top-p or top-k, and no system prompt. The derivative’s card overrides the base card. Every time.

Two traps stacked on top of that:

• Profiles override model defaults. A sampling fix applied to the model’s settings silently does nothing for any profile-suffixed endpoint. I now fix the profile first and the model second.
• Penalties did not disable the drafter. I assumed a non-neutral repetition penalty would trip the speculative path’s processor-conflict gate. Measured: it does not—that gate belongs to the native-MTP path on this runner. Assumptions about which knob disables which lane also need receipts.

And one sanctioned deviation, documented as an owner decision rather than a default: my reasoning lane runs sampling with thinking on and neutral penalties—off the derivative card’s greedy recipe—because the card’s repetition penalty exists as a greedy anti-looping fix and stops applying once sampling is on. That lane is the fastest measured config in §4, and its output was verified loop-free (longest repeated 6-gram across the test outputs: 2).

§7Housekeeping the tuning guides skip

• The disk cache shipped with a 185 GB ceiling. This Mac had ~153 GB free. An SSD prompt cache that defaults bigger than the disk it lives on is a full-disk incident on a schedule; I capped it at 40 GB with a 16 GB hot tier.
• Hide the helper models. Without that flag the 2B drafter is offered as a standalone chat model in every client—one accidental selection and your “27B review” is a 2B answering.
• The memory picture is calmer than the specs suggest. Both targets resident together hold 32.6 GB of 128 GB; the KV cache pages lazily at roughly 0.25 MB per token. Headroom is real, but it is headroom for context, not for skipping the math in deep dive 36.

The exact local receipt

This closes the “measurements pending” status from deep dive 33 and updates the current entry in the local model field notes. One honest asterisk: the measured stack is a pair of 4-bit community derivatives, not the stock 8-bit conversion the original receipt named—the download plan changed when the drafter did.

What I am not publishing: the exact repositories of the two modified community builds—deep dive 05 explains that boundary—and my local port and profile layout. The method transfers to any compatible target-and-drafter pair without either.

Primary sources

• Inco AI: the DFlash 2 drafter for Qwen3.8-27B — announcement and published speedups
• DFlash on GitHub — the drafter architecture and integration points
• Qwen: official Qwen3.8-27B model card — base sampling recipe and native context
• Deep dive 33: the download receipt this note settles