# Observer-first triage card

Version: 1.0
Updated: 2026-10-02
Method: https://suhailmohebi.com/notes/diagnose-the-observer-first/

Purpose: locate the layer that owns a failure before anyone changes a system that may not be broken.

Output: a short incident note with the observer, sideways results, failing layer, evidence window and an ownership decision.

## Use when

Someone reports “it’s down”, “it’s slow” or “the job failed”, and all you have is one viewpoint’s symptom.

## Quick start

1. Rewrite the report as one observation: who saw what, where, when, with the exact result.
2. Move sideways once (another device, network, account, region or client) before probing deeper.
3. Climb the read-only ladder and change nothing until one layer clearly owns the failure.

One failed viewpoint proves a symptom. It does not prove where the fault lives.

## 1. Failure claim and observer

| Field | Record |
|---|---|
| Claim as reported |  |
| Observer: person or system |  |
| Device, client, network and region |  |
| Identity: account, role or token scope |  |
| Time (UTC) |  |
| Exact request and exact result |  |

Weak: “The site is down.” Observation: “At 09:12 UTC a browser on laptop-2, on office Wi-Fi, timed out loading docs.example.com.”

## 2. Sideways matrix

> What is the smallest observer change that would make one entire class of causes impossible?

| Vantage | Result (exact) | Time (UTC) | If it works, suspect | If it also fails, suspect |
|---|---|---|---|---|
| Second client, same device |  |  | the first client’s state | the device or beyond |
| Second device, same network |  |  | the first device | the network or beyond |
| Same device, other network |  |  | the first network’s path | the device or the remote system |
| Other account or role |  |  | identity or permissions | something not tied to identity |
| Other region or remote vantage |  |  | one region’s path or edge | the remote system itself |

## 3. Layer ladder

All checks are read-only. Record each layer before moving outwards.

| Layer | Read-only check | Healthy looks like | Observed |
|---|---|---|---|
| Device link | interface up; signal, negotiated rate and local errors (OS network panel) | connected at the expected rate, no local errors |  |
| Local network | gateway and resolvers in use; `ping` the gateway; the system resolver’s answer; any proxy or VPN | expected settings; no loss; same answer as an outside resolver |  |
| Remote path | `dig` direct and via a second resolver; redirect target, TLS and timings with `curl`; pin one address; compare regions | expected code, usual timings, same answer everywhere |  |
| Application | the page after redirects carries the text the user needs; session, permission, data freshness | text present, right signed-in state, current data |  |
| Outcome | the person or downstream system confirms the result | the intended result received |  |

```bash
#!/usr/bin/env bash
# layer-probe.sh - one read-only check per layer, from the vantage point that runs it.
# Usage: bash layer-probe.sh HOST [TEXT_THE_USER_NEEDS] [SECOND_RESOLVER] [ADDRESS_TO_PIN]
set -u
host=${1:?usage: layer-probe.sh HOST [TEXT] [SECOND_RESOLVER] [ADDRESS_TO_PIN]}; text=${2:-}; resolver=${3:-}; pin=${4:-}
echo "checked_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ) host=$host"
echo "== 1 device link: not probed here; read interface, signal, negotiated rate and errors in the OS network panel"
echo "== 2 local network: gateway and resolvers in use, loss to the gateway (no reply can also mean ICMP is filtered), what applications resolve"
gw=$( { route -n get default 2>/dev/null || ip route show default 2>/dev/null; } |
  awk '/gateway:/ {print $2} /^default/ {for (i = 1; i < NF; i++) if ($i == "via") print $(i + 1)}' | head -n 1)
echo "gateway=${gw:-unknown}"
if command -v scutil >/dev/null 2>&1; then scutil --dns | grep 'nameserver\[' | sort -u; else grep '^nameserver' /etc/resolv.conf; fi
[ -z "$gw" ] || ping -c 5 -q "$gw" | tail -n 2
if command -v dscacheutil >/dev/null 2>&1; then dscacheutil -q host -a name "$host" | grep '^ip_address'; else getent hosts "$host"; fi
echo "== 3 remote path: what dig resolves, what a second resolver says, then the first HTTPS response"
dig +short "$host" A | sort
[ -z "$resolver" ] || { echo "-- via $resolver"; dig +short @"$resolver" "$host" A | sort; }
w='dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code} ip=%{remote_ip} redirect=%{redirect_url}\n'
curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -w "$w" "https://$host/"
[ -z "$pin" ] || { echo "-- pinned to $pin (skips this device's DNS)"; curl -sS --connect-timeout 10 --max-time 30 -o /dev/null -w "$w" --resolve "$host:443:$pin" "https://$host/"; }
echo "== 4 application: after redirects, lines with the text the user needs (0 = missing or fetch failed)"
[ -z "$text" ] || curl -sSL --connect-timeout 10 --max-time 30 "https://$host/" | grep -c -F -- "$text"
[ -z "$text" ] || [ -z "$pin" ] || { echo "-- pinned to $pin"; curl -sSL --connect-timeout 10 --max-time 30 --resolve "$host:443:$pin" "https://$host/" | grep -c -F -- "$text"; }
```

Needs `curl` and `dig`. Read the output with these limits in mind:

- Timings are cumulative: the TLS handshake took `tls - connect`; server wait is roughly `ttfb - tls`.
- `dig` asks a resolver directly; applications use the system resolver (`dscacheutil` on macOS, `getent hosts` on Linux). If they differ, the observer’s own resolver path is part of the story.
- No ping reply is not an outage: many hosts drop ICMP while serving HTTPS normally.
- A 200 can carry the wrong body: step 4 checks the text. The outcome layer is not a probe; confirm it with the person or the downstream record.

## 4. Evidence window

- First seen (UTC):
- Last known good (UTC) and its evidence:
- Changes inside the window (deploys, DNS, network, credentials, configuration):

| Time (UTC) | Observer | Observation | Evidence |
|---|---|---|---|
|  |  |  |  |

## 5. Ownership decision

> Change nothing until one layer is shown failing while the layer beneath it works, from at least two observers.

- Owning layer and owner:
- Smallest reversible change in that layer, and its rollback:
- Recovery check: the original observer plus one independent observer, on the user outcome.
- Decision: Change (you own it) / Hand over (someone else does) / Keep probing (ownership unclear)

## 6. Incident note

```text
VERDICT
[Most likely fault layer]; [current user impact].

OBSERVERS
[device / network / region / protocol / identity / time UTC]

POSITIVE EVIDENCE
[What worked, at which layer.]

NEGATIVE EVIDENCE
[What failed, with the exact response or measurement.]

NOT YET PROVEN
[Remaining causes and missing vantage points.]

NEXT PROBE
[Smallest read-only check that separates them.]
```

## Worked example

“The docs site is down”, reported at 09:12 UTC from laptop-2 on office Wi-Fi. Window: the DNS record moved to a new host at 08:30 UTC with a 300-second TTL; the old host went offline at 08:50; last good 08:47 (uptime check); first failure 08:52.

```text
VERDICT
Local network layer: the office resolver still serves the old address over 40 minutes after the move.
Office Wi-Fi users cannot load docs.example.com; everyone else can.

OBSERVERS
laptop-2, laptop-3, phone / office Wi-Fi, mobile data, hotspot / one region / HTTPS / no login / 09:12-09:25 UTC

POSITIVE EVIDENCE
Phone on mobile data and laptop-2 via hotspot load the page. Gateway 0% loss.
Pinned to the new address, laptop-2 on office Wi-Fi gets code=200 and the expected text.

NEGATIVE EVIDENCE
Office resolver returns the old address; a second resolver returns the new one.
With system DNS, curl on laptop-2 and laptop-3 ends in exit 28 (timeout).

NOT YET PROVEN
Why the resolver outlives the TTL: its cache policy or an upstream forwarder.

NEXT PROBE
Query the resolver's upstream forwarder directly: the new address there points at the office cache.
```

**Decision:** Hand over to the office network team, who own the cache flush; change nothing on the site or its DNS. Close only when laptop-2 and laptop-3 on office Wi-Fi load the page and show the expected text.

## Stop if

- Only one observer has been tested: find a second before naming a cause.
- The next step would restart, clear, purge, re-route or deploy before a layer owns the failure.
- The owning layer belongs to someone else (provider, network team, upstream service): hand over the note.
- A probe needs credentials or access you were not given.
- A component recovered but the original observer’s user outcome has not.

## Sources

- curl: man page, `--write-out` variables and `--resolve` — https://curl.se/docs/manpage.html (checked 2026-10-02)