# Local model test contract

Version: 1.0
Updated: 2026-10-02
Method: https://suhailmohebi.com/notes/local-model-field-notes/

Purpose: turn a local model trial into a dated record that another person can rerun and trust.

Output: one contract with an identity record, separate prefill, decode and end-to-end numbers, pass rates, usable context and an Adopt / Test / Drop verdict.

## Use when

You are about to rely on a local model, quantization, runtime or machine for real work, or to compare two local set-ups.

## Quick start

1. Name the job, its category and its pass condition before running anything.
2. Fill the identity record from files and logs, not memory, and hash the model file.
3. Measure prefill and decode separately at stated depths, with repeats, then grade your own tasks several times.

A tokens-per-second figure without its denominator, depth and cache state is not a measurement.

## 1. Job and category

> This [on-device / private / open-weights] set-up will do [job] on [input shape: typical and largest prompt in tokens].
> It passes if [measurable condition] across [n] runs, within [latency limit].

| Category | Means | Does not promise |
|---|---|---|
| On-device | Runs on the machine in front of you | Frontier capability or long context |
| Private | Runs on infrastructure you control | One-machine simplicity |
| Open weights | Weights can be inspected or deployed | Affordable local inference or identical provider behaviour |

## 2. Identity record

| Field | Record | Read from |
|---|---|---|
| Date, hardware, memory, OS and drivers |  |  |
| Runtime and exact version or commit |  |  |
| Model file, source and SHA-256 |  |  |
| Quantization: format, bits, group size |  |  |
| Configured context; KV cache type for K and V |  |  |
| Batch, micro-batch, parallel slots |  |  |
| Sampling settings as applied: request body or server log |  |  |
| Speculative decoding: draft model or none |  |  |
| Every other non-default flag; other services resident |  |  |

```bash
# Read-only. Linux: use sha256sum instead of shasum -a 256.
MODEL=/models/model.gguf   # Split models: hash every shard.
shasum -a 256 "$MODEL"
ls -l "$MODEL"
uname -srm; sw_vers -productVersion 2>/dev/null
```

## 3. Speed protocol

- Prefill (reading the prompt) and decode (writing the answer) differ by large factors. Report them separately, each with its prompt length and depth.
- Repeat each configuration (three runs or more); report median and min–max, or label it mean ± standard deviation.
- Change one variable per comparison.

```bash
# pp = prefill, tg = decode; three depths x 5 repetitions; JSONL keeps every finished sample.
# Set -fa, -ctk and -ctv as your server runs them.
llama-bench -m /models/model.gguf -p 512,4096 -n 128 -d 0,8192,32768 \
  -fa on -ctk f16 -ctv f16 -r 5 -o jsonl > bench-01.jsonl
python3 -c 'import json, statistics, sys
print("test depth median min max runs")
for r in map(json.loads, open(sys.argv[1])):
    p, g, s = r["n_prompt"], r["n_gen"], r["samples_ts"]
    t = "pp%d" % p if not g else "tg%d" % g if not p else "pp%d+tg%d" % (p, g)
    print(t, r["n_depth"], round(statistics.median(s), 1), round(min(s), 1), round(max(s), 1), len(s))' bench-01.jsonl
```

The JSON also records `build_commit`, `type_k`, `type_v` and `n_batch`. llama-bench timings exclude tokenisation and sampling, follow a warm-up unless you add `--no-warmup`, and `pp512 @ d8192` times 512 new tokens on a filled 8,192-token cache: a rate at depth, not the time to read an 8k prompt.

Real tasks, streamed through the server you will use:

- TTFT (time to first token), cold and warm. Confirm warm reuse by the wall-clock drop, not a cache counter.
- Decode rate = (completion tokens − 1) ÷ (wall time − TTFT).
- End-to-end rate = completion tokens ÷ total wall time. Never compare it with a decode baseline; compare warm runs end-to-end only.

| Measure | Prompt or depth | Cache | Median | Spread | Runs |
|---|---|---|---|---|---|
| Prefill, synthetic |  | n/a |  |  |  |
| Decode, synthetic |  | n/a |  |  |  |
| TTFT, real task |  | cold / warm |  |  |  |
| Decode rate, real task |  |  |  |  |  |
| End-to-end, real task |  |  |  |  |  |

## 4. Quality protocol

1. Take 5–20 tasks from the real job, each graded in code or by a written rubric.
2. Prove the grader first: it must pass known-good and fail known-bad outputs. Disclose any change that makes passing easier.
3. Run every task at least three times. Greedy decoding is not guaranteed to repeat across prompt-cache states; one run is one sample.
4. Run the same tasks and grader on the cloud model you would otherwise use if the data may leave; else record “cloud: not run”.

| Task | Check | Passes / runs | Cloud passes / runs | Observed failure |
|---|---|---|---|---|
|  |  |  |  |  |

Cover recall, tool calls or structured output, and instruction following; recall and tool calls slip first at low precision.

## 5. Usable context

| Limit | How measured | Tokens |
|---|---|---|
| Configured window | runtime setting or start-up log |  |
| Fits in memory | largest context that loads and runs with headroom |  |
| Acceptable latency | largest prompt whose cold TTFT meets your limit |  |
| Reliable | deepest prompt where recall, rule and format checks still pass |  |
| Usable context | the smallest of the four |  |

## 6. Resource record

- Peak memory during the deepest test, and the tool that reported it:
- Swapping, memory pressure or throttling seen (yes / no):
- Power, thermals, noise: measured, or “not measured”:

## 7. Comparability rules

- [ ] Exactly one variable differs between the two runs.
- [ ] Same model hash, quantization, runtime version, context, KV cache type, batch and concurrency.
- [ ] Same sampling, prompt shape, depth and cache state.
- [ ] Same metric definition. Acceptance rate and mean accepted length, or reasoning-trace speed and answer speed, never share a column.
- [ ] Anything else that differs is named next to the result.

## 8. Verdict

- Adopt: every pass condition holds at the usable context across repeated runs.
- Test: a key number is a single run, or a check is unproven.
- Drop: the job needs the property that failed.

## Worked example

Illustrative numbers, not a benchmark.

```text
DATE / MACHINE / RUNTIME: 2026-10-02 / 128 GB unified-memory laptop / llama.cpp commit abc1234
MODEL: 27B-parameter open-weight model, Q4_K_M GGUF, SHA-256 recorded; category: on-device
JOB: private review of design notes, prompts ~6,100 tokens
PASS: every check in >= 90% of task runs over 3 runs, every planted contradiction found, cold TTFT <= 60 s
CONTEXT: configured 32,768; fits 32,768; latency OK at 16,384 (cold TTFT 44 s); reliable at 16,384 (deepest tested)
SYNTHETIC (5 runs): pp4096 @ d0 410 t/s (402-415); tg128 @ d0 24.1 t/s (23.8-24.4)
  pp512 @ d8192 340 t/s (333-346); tg128 @ d8192 22.0 t/s (21.8-22.3)
REAL TASK: cold TTFT 15.2 s, warm 1.6 s; decode 22.6 t/s; 520 tokens in 38.2 s (13.6 t/s end-to-end); warm 520 tokens in 24.6 s (21.1 t/s)
PEAK MEMORY: 26 GB of 128 GB, no swapping
QUALITY: 34/36 task runs passed every check (12 tasks x 3 runs); cloud: not run (private); all contradictions found
FAILURE: the citation-format check failed 2 of 36 times, on short and long prompts alike: a habit, not depth
NOT MEASURED: concurrency above 1, power
DECISION: ADOPT up to 16,384 tokens; re-test on a new model file, quantization, runtime or machine
```

The 13.6 t/s end-to-end figure is not a regression against 24.1 t/s decode: prefill fills 15 of the 38 seconds.

## Stop if

- The model hash, runtime version or a flag no longer matches the identity record.
- A number lacks its denominator, depth or cache state.
- The grader fails a known-good output or passes a known-bad one.
- The decision rests on single runs.
- Swapping, memory pressure or throttling appears mid-run.
- A comparison needs two runtimes’ metrics in one column without matching definitions.

## Sources

- ggml-org/llama.cpp: llama-bench README — https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md (checked 2026-10-02)
- ggml-org/llama.cpp: llama-bench source — https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/llama-bench.cpp (checked 2026-10-02)