Why local LLM benchmark charts disagree

Research checked 31 October 2026. Compiled from published tests. No hardware benchmarks were run for this article; measurements belong to the credited testers.

The published record offers examples of all three.

A cached prompt is a different workload

Federico Viticci’s MacStories review reports 50.0 seconds to the first token for a cold, approximately 128K-token Flash-Next prompt on a 256GB M5 Ultra. Repeating the prompt with the cache warm took 3.3 seconds. Both measurements belong to the same system. Quoting either as its universal response time would mislead readers. Viticci’s time-to-first-token tests.

For a document uploaded for the first time, the cold measurement is relevant. For an ongoing conversation that can reuse previous processing, the cached measurement better reflects that work. A reader needs to know which situation the chart describes.

Total throughput does not describe one person’s wait

LMSYS’s October 2025 Spark testing reports Llama 3.1 8B generation throughput of 20.5 tokens per second at batch size one and 368 at batch size 32 under SGLang. The larger number is an aggregate serving result. It does not show that a single chat runs at 368 tokens per second. Jerry Zhou and Richard Chen, LMSYS.

A service supporting many users can benefit from that throughput. An individual using a coding assistant still needs first-response delay, per-request speed and completion time.

Even the token counter can be wrong

Puget Systems disclosed that its benchmark harness used the wrong tokenizer for DeepSeek-R1-Distill-Llama-8B. Its September correction lowered the R9700 single-user result from 61.7 to 34.5 tokens per second. Repeating the original figure after that correction would preserve an error, even with a source link. Puget’s correction notice.

A familiar label may hide different model files

Ollama’s Spark article notes that some files distributed elsewhere under the MXFP4 label further quantize attention layers, whereas its tested distribution retains BF16 in those layers. Matching a model name and a broad quantization label therefore may still leave meaningful differences. Ollama’s methodology note.

Before putting two scores beside each other, check the exact model and quantization, runtime version, input length, output length, cache state and concurrency. Also check whether speculative decoding was enabled and whether the score includes reasoning tokens.

Where those details differ, keep the results in separate tables and explain what each establishes. A chart can still be informative without naming an overall winner. Publishing the conditions alongside the numbers lets readers decide whether the test resembles the work they actually need done.

Read the series

• Mac Studio and RTX cards: reading published LLM tests
• DGX Spark 64GB: what published tests can tell us
• AMD local LLM tests: the configuration behind the score
• Why local LLM benchmark charts disagree (this article)
• Local AI cost calculations: power, usage and payback

For memory planning, use the model-fit guide. Browse all Local AI notes.

Suhail Mohebi · published 31 October 2026 · the cover is a generated illustration