# Agent incident review

Version: 1.0
Updated: 2026-10-02
Method: https://suhailmohebi.com/notes/when-success-is-still-broken/

Purpose: turn an agent failure into fixes aimed at the layer that failed, without blaming a person or “the AI”.

Output: a blameless review with a UTC timeline, the agent’s quoted inference, root-cause classes and verifiable action items.

## Use when

An agent reported a success that was not true, or took or skipped an action it should not have, and the result reached users, data or a release.

## Quick start

1. Contain first, then preserve the transcript, tool-call log, configuration and permissions as they were.
2. Fill sections 1–6 from those records, quoting the agent verbatim; never reconstruct from memory.
3. Classify the root cause, then give every action item an owner, a due date and a verification.

“Fix the bug” and “improve the prompt” are not action items: name the layer, the change and the check that proves it.

## 1. Summary and impact

Blameless: assume every person and agent acted reasonably on the information and permissions they had, and ask why those were wrong. Name roles, not people.

- Summary, in two sentences:
- Impact: who or what, how much, for how long:
- Detection: monitor, user report, reviewer or chance; time to detect:

## 2. Timeline (UTC)

| Time (UTC) | Actor | Event | Evidence |
|---|---|---|---|
|  |  |  |  |

Include the trigger, each consequential action, the success signal, detection, containment and resolution. To put commits on the same clock:

```bash
TZ=UTC git log --all --reverse --since='2026-10-01T00:00:00Z' --until='2026-10-02T23:59:59Z' \
  --date=format-local:'%Y-%m-%dT%H:%M:%S%z' --format='%cd %h %cn %s'
```

Give `--since` a time: git fills a bare date with the current time of day. The filter, `%cd` and `%cn` all use the committer. Every time should end +0000; another offset means `TZ=UTC` was lost.

## 3. Outcome ladder

| Rung | Question | Signal that fired | Real outcome |
|---|---|---|---|
| 1 Transport | Did the request reach something? |  |  |
| 2 Acceptance | Did the receiving system accept it? |  |  |
| 3 Completion | Did every required step finish? |  |  |
| 4 State | Did the authoritative record change? |  |  |
| 5 Experience | Can a person observe the expected result? |  |  |
| 6 Live truth | Does production serve that result now? |  |  |

Mark the rung the “done” rested on and the highest rung proven; the gap is the incident. Agents overclaim by checking that their step finished, not that the outcome is true.

## 4. Agent context at the time

- Agent identity or role, and what started it:
- Instruction files and their version (commit or hash):
- Runtime configuration and its version:
- Permissions and credentials in scope (names, never values), and recent changes:
- Tools available, and missing tools that would have shown the truth:
- Policy decisions: allowed or denied, under which policy version:

## 5. Tool-call chain

| # | Time (UTC) | Tool and input | Output | Policy decision | What it proved |
|---|---|---|---|---|---|
| 1 |  |  |  | Allowed / Denied / None |  |

Copy from immutable logs. Write “not logged” for any gap; never fill one from memory.

## 6. What the agent inferred

> Verbatim from the transcript: “[quote]”

- The correct inference would have been:
- The signal it missed:
- The context or tool change that would have produced the correct inference:

## 7. Root-cause classification

| Class | Test | Fix lives in | Contributed? |
|---|---|---|---|
| Agent | Wrong plan, or a claim beyond what its context and tools could show | Eval cases, autonomy threshold | Yes / No |
| Tool | A tool returned wrong, partial or manipulated data, or did not surface the field that mattered | The tool; security if manipulated | Yes / No |
| Data | The context was stale, incomplete or corrupted | Retrieval, cache, system of record | Yes / No |
| Human–agent handoff | Escalation at the wrong threshold, or an approval that should have been refused | Escalation policy, reviewer’s view | Yes / No |
| Policy or permission | Access broader than intended, or a blocking policy missing or disabled | Platform or security owner | Yes / No |

An incident often involves more than one class; marking each one shows trends across incidents.

## 8. Contributing factors

- Human: [an approval given without evidence, an ambiguous brief]
- Process: [no acceptance contract, no independent review]
- Monitoring: [the signal that would have caught it sooner, and why it did not]

## 9. Action items

| Type | Action | Owner | Due | Verified by |
|---|---|---|---|---|
| Guardrail or check |  |  |  |  |
| Eval case added |  |  |  |  |
| Autonomy threshold |  |  |  |  |
| Monitor change |  |  |  |  |
| Decision-ledger entry |  |  |  |  |

Eval items commit to a measured pass rate on a named eval; threshold items name a measurable target. Close an item only when its verification was observed.

Follow-up review date: [date]. Ask: did the same class recur, and did the named eval’s failure rate fall?

## Worked example

**Summary:** agent-a changed a feedback-form handler, got 200 from one test request and reported “done”. Storage rejected the new payload: 37 submissions in 19 hours were lost.

| Time (UTC) | Actor | Event | Evidence |
|---|---|---|---|
| 2026-10-01 07:40 | agent-a | Deploys the handler change, as policy allows | Deploy log |
| 2026-10-01 07:42 | agent-a | One test request returns 200; reports “done” | Transcript |
| 2026-10-02 02:40 | ops owner | Notices the daily export is empty | Export file |
| 2026-10-02 03:05 | ops owner | Rolls back; stored submissions resume | Storage records |

**Context:** instructions at `abc1234`; form-handler deploys allowed without review; no tool could read stored records or the worker log.

**Ladder:** “done” rested on rung 1. Rung 2 held (handler log: payload accepted); rungs 3–6 failed.

> “The endpoint returns 200 with the new payload, so the change is working end to end.”

**Correct inference:** 200 proves transport only; the write happens later. **Missed signal:** the storage error in the worker log, which the agent could not read.

**Classes:** agent (took transport for completion), tool (nothing could read stored state), human–agent handoff (a consequential deploy with no escalation).

| Type | Action | Owner | Due | Verified by |
|---|---|---|---|---|
| Guardrail | Handler returns an error when the write fails | form owner | 2026-10-09 | Test fails before, passes after |
| Eval case | Add this trace: check stored state before “done” | agent owner | 2026-10-16 | `done-needs-state` passes 19 of 20 runs |
| Autonomy threshold | Handler deploys need human approval of a state proof until the eval holds | release owner | 2026-10-05 | Policy diff; next 10 deploys |
| Monitor change | Alert on zero stored submissions for 2 daytime hours | ops owner | 2026-10-09 | A drill fires the alert |
| Decision-ledger entry | “Done” names its state-rung proof before work starts | team lead | 2026-10-05 | Entry linked from the instructions |

Follow-up review: 2026-11-02.

## Stop if

- Logs or transcripts are missing or edited: add a logging action; never reconstruct the agent’s reasoning.
- The agent still runs with the same permissions while its failure class is open: contain it first.
- A credential may be exposed or a tool compromised: hand the incident to the security owner first.
- The review blames a person, or an action item lacks an owner, a date or a verification.

## Sources

- Trussed AI: AI Agent Incident Postmortem Template — https://trussed.ai/resources/ai-agent-incident-postmortem-template-and-guide (checked 2026-10-02)
- Tian Pan: The Postmortem Template With No Row for the Model’s Inference — https://tianpan.co/blog/2026/06/01/the-postmortem-template-with-no-row-for-the-models-inference (checked 2026-10-02)
- Google SRE book: Postmortem Culture: Learning from Failure — https://sre.google/sre-book/postmortem-culture/ (checked 2026-10-02)