# Agent eval set starter

Version: 1.0
Updated: 2026-10-02
Method: https://suhailmohebi.com/notes/tool-calling-at-250k-context/

Purpose: measure whether an agent still does its job after a change, with graders you have tested and results you can repeat.

Output: a small task set with validated graders, a results record with pass rates, and a ship or hold decision.

## Use when

Before you change the model, prompt, tools, skills, context size or harness behind an agent that does real work, or after an incident you never want to repeat.

## Quick start

1. Define the job, then write tasks from real work and real failures, including tasks where the agent should not act.
2. Run every grader on a known-good and a known-bad output before trusting any score.
3. Run several trials per task, read the failed transcripts, and apply the decision rule in section 8.

Grade the grader before trusting the score, and disclose every grader fix that made passing easier next to the number it changed.

## 1. Job definition

> This agent [does what] for [whom] using [tools]. Success means [observable end state]. It must never [forbidden action].

- Outcome: check the final state in the environment (record exists, file changed, test passes), not the agent’s report that it happened.
- Standing rules: rules every task must obey, such as an audit ID on every tool call. State them once at the top of the context; grade them on every task.

## 2. Task table

Prefix regression tasks (must keep passing) with R and capability tasks (may start low) with C.

| ID | Input and setup | Expected outcome | Grader | Pass condition |
|---|---|---|---|---|
| R1 |  |  | Code / Model / Human |  |
| C1 |  |  | Code / Model / Human |  |

- [ ] Draw tasks from real failures and the checks you already run by hand; 20–50 tasks is a strong start.
- [ ] Balanced: include tasks where the behaviour should not happen, such as no tool call when none is needed.
- [ ] Unambiguous: two experts would agree on pass or fail, and everything the grader checks is stated in the task.
- [ ] Solvable: each task has a reference solution that passes its graders.
- [ ] Graders: code first (exact match, state check, tool-call arguments); a model rubric only for judgement, calibrated on human labels, allowed to say “Unknown”.
- [ ] Grade the outcome, not the path, unless the path is the rule (for example, confirm before sending).

## 3. Validate the graders first

| Grader | Known-good output | Known-bad output | Good passes? | Bad fails? |
|---|---|---|---|---|
|  |  |  | Yes / No | Yes / No |

A grader that fails a known-good output or passes a known-bad one is broken. Fix it before reading any score.

A task that fails every trial on every configuration more often means a broken task or grader than an incapable agent.

## 4. Trials and nondeterminism

- Run n trials per task, each from a clean state: no files, caches or history left by earlier trials. Use at least three.
- Report c/n per task, never one run. Temperature 0 is not determinism: cache state and batching can flip near-tied outputs.
- pass@k is the chance that at least one of k trials succeeds; pass^k is the chance that all k succeed.
- From n ≥ k trials with c passes, estimate pass@k = 1 − C(n−c, k) / C(n, k) and pass^k = C(c, k) / C(n, k) (C is the binomial coefficient), then average over tasks.
- With independent trials at 0.75 success, pass@3 ≈ 0.98 but pass^3 ≈ 0.42. Use pass^k when users need it to work every time.
- Separate failures that move with the variable you changed from marginal habits that fail at random everywhere.

```bash
# Suite pass rate, pass@k and pass^k from passes per task (needs Python 3.8+).
N=4; K=3; PASSES=(4 4 4 4 3)   # trials per task, k, passes for each task
python3 - "$N" "$K" "${PASSES[@]}" <<'EOF'
import sys
from math import comb
n, k, *cs = map(int, sys.argv[1:])
assert cs and 0 < k <= n and all(0 <= c <= n for c in cs), "need tasks, 0 < k <= n, 0 <= c <= n"
t = len(cs)
print(f"pass rate {sum(cs) / (n * t):.2f}, pass@{k} {sum(1 - comb(n - c, k) / comb(n, k) for c in cs) / t:.2f}, "
      f"pass^{k} {sum(comb(c, k) / comb(n, k) for c in cs) / t:.2f}")
EOF
```

## 5. Compare fairly

- [ ] Change one variable at a time: model, prompt, tools, context size, decoding settings or harness.
- [ ] Same tasks, same grader version, same trial count, same environment.
- [ ] If a grader changes, re-run the baseline with it before comparing.
- [ ] To test context size, insert neutral padding between the standing rules and the request and run the same tasks at each depth.
- [ ] Re-run the baseline once; a difference smaller than that run-to-run spread cannot be told from noise.

## 6. Read the transcripts

- [ ] Read every failed trial and a sample of passes.
- [ ] Label each failure: agent error, grader error, task ambiguity or environment. Only agent errors count against the agent.
- [ ] Check passes for the wrong reason: a forbidden path, a grader loophole or state left by another trial.

## 7. Turn incidents into regression tasks

- [ ] Reproduce the incident input with the minimum setup, and write the pass condition as the outcome the user missed.
- [ ] Confirm the task fails on the version that caused the incident, or on a known-bad output, and passes after the fix.
- [ ] Add it to the R set. Regression tasks must pass every trial; a capability task joins them once it does.

## 8. Results and decision

| Run | Variable changed | Tasks × trials | Grader version | Trials passed; pass^k | R failures |
|---|---|---|---|---|---|
| Baseline | none |  |  |  |  |
| Candidate |  |  |  |  |  |

**Ship** only if all four hold; otherwise **hold** and record which one failed:

- every R task passed every trial;
- the C pass rate is within the baseline’s run-to-run spread, or better;
- every failure has been read and labelled;
- the grader version matches the baseline’s.

## Worked example

Variable: the system prompt. The candidate replaces a 2,400-word prompt with a 900-word version that keeps the standing rules.

| ID | Input and setup | Expected outcome | Grader | Pass condition |
|---|---|---|---|---|
| R1 | “Weather in Lisbon tomorrow?” Rule: trace_id X9-441 on every call | One weather call | Code | One call; city matches; trace_id present |
| R2 | “Email the summary to the team.” No recipient | Asks for the recipient | Code | No send call; reply requests the recipient |
| R3 | “What is 2 + 2?” | Answer, no tool, no preamble | Code | Zero tool calls; no “Sure” or “Certainly” |
| R4 | “Convert 1,250.50 EUR to USD.” | Amount sent as a number | Code | `amount` is the number 1250.5 |
| C1 | Summarise an incident log for on-call staff | Cause, impact, next step | Model rubric, calibrated on 10 human labels | Each criterion 4 of 5 or more |

Grader check: R2 failed the known-good reply “Please confirm the recipient.” because it demanded a question mark. Loosened to accept any request for the recipient; disclosed below.

| Run | Variable changed | Tasks × trials | Grader version | Trials passed; pass^3 | R failures |
|---|---|---|---|---|---|
| Baseline | none | 5 × 4 | g2 (R2 loosened) | 19/20; 0.85 | 0 |
| Baseline re-run | none | 5 × 4 | g2 | 18/20; 0.80 | 0 |
| Candidate | trimmed prompt | 5 × 4 | g2 | 18/20; 0.80 | 0 |

Passes per task (R1–R4, C1, of 4): baseline 4 4 4 4 3; re-run 4 4 4 4 2; candidate 4 4 4 4 2. The section 4 calculator reproduces each row.

Transcripts: all failures are C1 missing the next step in both prompt versions, a marginal habit rather than damage from the trim.

**Decision:** ship. All four conditions hold; C1 passed 2 of 4 on the candidate, inside the baseline’s own 2–3 of 4.

## Stop if

- A grader passes a known-bad output or fails a known-good one.
- The runs you compare differ in tasks, graders, trial count, or more than one variable.
- Failures have not been read and labelled.
- Trials share state, so they are not independent.
- A grader was loosened and the change is not disclosed next to the score.

## Sources

- Anthropic: Demystifying evals for AI agents — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents (checked 2026-10-02)
- Yao et al.: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — https://arxiv.org/abs/2406.12045 (checked 2026-10-02)