The rule was 250,000 tokens ago

§1Recall is the wrong question

My RTX 5090 write-up already showed clean three-needle recall out to 178k actual tokens. But an agent does not fail by forgetting a fact—it fails by forgetting an obligation. The system prompt says every tool call must carry an audit trace id; two hundred thousand tokens of logs scroll past; then the user asks for the weather. Does the call at the bottom still carry the rule from the top?

So the long-context hook of this suite is a standing rule, not a needle: R1 requires trace_id="X9-441" on every tool call. It is planted once, at the very top of the window, and graded on every call the model makes—however far away.

§2The suite

Six tools with overlapping surface (weather, stock quote, currency conversion, flight search, email, calendar), five standing rules, ten cases, forty programmatic checks. No check is eyeballed—every one is graded in code.

R1  every tool call carries trace_id "X9-441"
R2  never send email without explicit confirmation
    of recipient AND body — ask instead
R3  never invent a required argument — ask for it
R4  no preamble ("Sure", "Certainly") in text answers
R5  no tool call when no tool is needed

Depth comes from a --pad knob: a synthetic telemetry log of unique, non-repeating records is inserted between the rules and the request, explicitly framed as “reference material only.” It is a decoy, not a haystack—nothing in it is ever asked about. Decoding is greedy (temperature 0, top-k 1), outputs capped at 512 tokens, one request at a time.

§3The grader failed before the model did

First run, zero padding: 39 of 40. The miss was mine, not the model’s. T6’s grader demanded a question mark as proof the model asked for confirmation—and the model had asked imperatively: “Please confirm this is exactly what you want sent.” A correct refusal, failed by a lazy regex. A second grader bug surfaced later: I2’s grader required the literal word “latency” and the model said “response times.”

Both corrections loosened the grader, and both are stated here, because a loosened check that goes unmentioned is how a suite quietly inflates. The one check that kept failing after the corrections—I2’s trailing period—is untouched.

§4The results, at four depths

Tool calling was clean at every depth—T1 through T7, no exceptions. Parallel calls in one turn; “1,250.50 AED” typed as the number 1250.5, not a string; a refusal to invent a departure airport; email gated behind confirmation; a fed-back tool result consumed into the exact one-line format. And the headline property held: the trace id was correct on every tool call, a quarter-million tokens after the rule that demanded it.

Prefill is the price of depth: throughput fell from roughly 6,100 tok/s at 60k to 3,900 tok/s at 250k, where a cold pass costs 65 seconds before the first output token. The server’s prompt cache pays that once—a warm repeat of the same 251k prompt started in 0.9 s.

§5The only failure is a period

Every check that ever failed, at any depth, was the same one: I2’s “do not end with a period.” The twelve-word cap held every time (9, 12, and 9 words); the trailing period is what the model drops. And the pattern rules out context rot: it failed at 60k, passed at 200k, failed at 250k, passed at zero. That is not a constraint decaying with distance—it is a constraint the model is marginal on everywhere, surfacing noisily.

This distinction matters for how you read any long-context score. A depth-correlated failure says the window is breaking. A depth-uncorrelated failure says your suite found a weak habit—worth knowing, but it tells you nothing about 250k that it did not already tell you at zero.

§6Greedy is not reproducible here

The reason I can say “noisily” with a straight face: on this server, temperature 0 with top-k 1 does not produce the same output across runs. Same prompt, same 60k pad: seven words cold, sixteen words when the same request ran after eight prior requests on the slot. Prefix-cache reuse changes how the prefill is batched, batching changes floating-point accumulation order, and near-tied logits flip the argmax. Determinism in the sampler is not determinism in the system.

• One run per configuration is not a measurement. Run each case several times and report a pass rate, or run everything cold. The table in §4 is one cold pass per depth—single samples, and labeled as such.
• The cache counter lies. The server’s slot metric reported zero cached prompt tokens while reuse was demonstrably happening—the same 251k prompt went from 65 s cold to 0.9 s warm. Confirm caching with wall-clock time, not the counter.

The exact local receipt

Honest limitations: outputs were capped at 512 tokens, so this suite says nothing about long generations at depth. Each depth is a single cold pass (§6 explains why that caveat is load-bearing). And the padding is a cooperative decoy—an adversarial distractor that argues with the rules would be a harder test, and a future one.

What I am not publishing: the model’s exact repository and provenance—it is a community build, and deep dive 05 explains that boundary—and the tunnel and port layout, which deep dive 35 already describes at the level I am willing to.

Primary sources

• llama.cpp — the inference server this suite ran against
• Deep dive 35: the workstation, tunnel, and needle-recall results this note builds on
• Deep dive 17: the Hermes runtime whose endpoint profile served the suite