# Policy-as-a-test kit

Version: 1.0
Updated: 2026-10-02
Method: https://suhailmohebi.com/notes/policy-as-a-test/

Purpose: turn a written decision into a check that fails at the moment a person or an agent undoes it.

Output: a rules file and check script wired into pre-commit, CI and the release gate, plus a blind-spots list and a named reviewer.

## Use when

A decision lives only in a brief or context file (a retired endpoint, a removed flag, a renamed feature) and new sessions keep reintroducing it.

## Quick start

1. Write each decision as an invariant that a file or build output can visibly violate.
2. Put exact strings in the rules file and run the check where changes land: pre-commit, CI and the release gate.
3. List what the check cannot see and name who reviews that part.

A rule that cannot fail is a preference. A green check is evidence, not proof.

## 1. Decision to invariant

| ID | Decision | Testable invariant | Check pattern | Runs at |
|---|---|---|---|---|
| D-014 | The v1 export endpoint is retired | No published file contains `/api/v1/export` | forbidden term | pre-commit, CI, release |
| D-015 | The `--legacy-auth` flag was removed | No doc or example shows `--legacy-auth` | forbidden term | pre-commit, CI, release |
| D-016 | Every page declares its canonical URL | Each built HTML file contains `rel="canonical"` | required term | release |
| D-017 | One registry entry per template | file count equals entry count | count or parity | CI |
|  |  |  |  |  |

Encode what is exact: strings, routes, files, counts and structure. Intent, tone and framing go to a reviewer (section 6).

## 2. Failure mode

> A violation of [ID] looks like [exact string, missing element or mismatched count] in [source files or build output].

If you cannot complete it, the decision is not testable yet: narrow it, or keep it as a review question.

## 3. Rules file and check

Each rule documents itself: what is forbidden, why, and what to do instead. Keep the file outside the scanned path, or the gate flags its own rules.

```text
# ID | forbidden text (fixed string, case-insensitive) | because | instead
D-014 | /api/v1/export | the v1 export endpoint was retired | link to /api/v2/exports
D-015 | --legacy-auth | the flag was removed | show the token login steps
```

```bash
#!/usr/bin/env bash
# policy-check.sh - fail when text that a recorded decision retired comes back.
# Usage: bash policy-check.sh RULES_FILE PATH_TO_SCAN
# Rules: one per line, "ID | forbidden text | because | instead"; lines starting with # are notes.
# Matching: fixed string, case-insensitive, text files only; .git/ is skipped.
# Exit: 0 pass, 1 a decision was violated, 2 the check could not run (never a silent pass).
set -u
rules=${1:-}; target=${2:-}
[ -f "$rules" ] && [ -r "$rules" ] && [ -e "$target" ] || { echo "ERROR: need a readable RULES_FILE and an existing PATH_TO_SCAN" >&2; exit 2; }
[ -n "$(grep -rlI --exclude-dir=.git -e '' -- "$target" | head -n 1)" ] || { echo "ERROR: no text files to scan in $target" >&2; exit 2; }
trim() { local s=$1; s=${s#"${s%%[![:space:]]*}"}; printf '%s' "${s%"${s##*[![:space:]]}"}"; }
count=0; failed=0
while IFS='|' read -r id text because instead <&3 || [ -n "$id" ]; do
  id=$(trim "$id"); case $id in ''|'#'*) continue ;; esac
  text=$(trim "${text:-}"); because=$(trim "${because:-}"); instead=$(trim "${instead:-}")
  [ -n "$text" ] && [ -n "$because" ] && [ -n "$instead" ] ||
    { echo "ERROR: rule $id needs forbidden text, because and instead" >&2; exit 2; }
  count=$((count + 1))
  files=$(grep -rlIiF --exclude-dir=.git --regexp="$text" -- "$target"); status=$?
  [ "$status" -le 1 ] || { echo "ERROR: scan failed for rule $id" >&2; exit 2; }
  [ -n "$files" ] || continue
  failed=$((failed + 1))
  while IFS= read -r f; do
    grep -niF --regexp="$text" -- "$f" | cut -d: -f1 | while IFS= read -r n; do
      echo "$f:$n: [$id] found \"$text\" - because $because; instead $instead"
    done
  done <<EOF
$files
EOF
done 3< "$rules"
[ "$count" -gt 0 ] || { echo "ERROR: no rules found in $rules" >&2; exit 2; }
echo "NOTE: this gate matches strings, not intent. A clean pass is not a judgement that the meaning is acceptable."
[ "$failed" -eq 0 ] || { echo "FAIL: $failed of $count rules violated in $target"; exit 1; }
echo "PASS: $count rules, no forbidden text in $target"
```

## 4. Other check patterns

Run these inside a check script; each exits 1 on a violation, and the first exits 2 when there is nothing to check. Count and schema checks need `jq`.

```bash
# Required term: every built page declares a canonical URL; prints the pages that do not.
[ -n "$(find dist -name '*.html' 2>/dev/null | head -n 1)" ] || { echo "ERROR: no HTML pages in dist" >&2; exit 2; }
missing=$(grep -rL --include='*.html' -F 'rel="canonical"' dist); [ -z "$missing" ] || { echo "$missing"; exit 1; }
# Count or parity: one registry entry per template file.
[ "$(find content/templates -name '*.md' | wc -l)" -eq "$(jq length content/templates.json)" ] || exit 1
# Schema: every registry entry has the fields the page needs (jq -e fails on false or null).
jq -e 'all(.[]; has("id") and has("url") and has("version"))' content/templates.json > /dev/null || exit 1
```

An empty registry passes the schema check (`all` of nothing is true); pair it with the count check.

## 5. Where to run it

| Stage | What it guarantees |
|---|---|
| Pre-commit hook | early warning only: local, reads the working tree rather than the staged snapshot, and skipped by `git commit --no-verify` |
| CI, as a required job | a violation cannot merge into the shared branch |
| Release gate, on the built output | generated pages, feeds, sitemaps and metadata are covered too |

Install the hook (changes local state: writes `pre-commit` into the hooks directory that every worktree of this repository shares, or into `core.hooksPath` if set; it refuses to replace an existing hook, so merge into that one by hand):

```bash
hooks=$(git rev-parse --git-path hooks) && mkdir -p "$hooks" &&
( set -C; printf '#!/bin/sh\n[ -f policy/policy-check.sh ] || { echo "pre-commit: no policy/ on this branch; not checked" >&2; exit 0; }\nexec bash policy/policy-check.sh policy/rules.txt site\n' > "$hooks/pre-commit" ) &&
chmod +x "$hooks/pre-commit"
```

Git runs hooks from the top of the work tree and ignores hooks that are not executable.

## 6. Blind spots and the paired judgement

The check cannot see:

- the same idea in other words, another language or an image;
- text split across lines or markup, or encoded (`%2F`, HTML entities);
- binary files, PDFs and screenshots, which are skipped;
- anything outside the scanned path: deploy-time output, embeds, caches, other repositories, git history.

Paired judgement, recorded per release:

- Reviewer (person or adversarial agent):
- Reads for: retired decisions restated in other words, intent, tone and framing.
- Verdict and reason: Pass / Fail

## 7. Review or expiry

- Owner of each rule:
- Review when: the decision changes, someone asks to weaken a rule, a false positive appears, or [date].
- Retire a rule with its decision: a gate guarding a reversed decision teaches people to bypass gates.

## Worked example

Under D-014, an agent drafting a tutorial links the old endpoint. The pre-commit hook stops the commit:

```text
site/docs/guide.md:3: [D-014] found "/api/v1/export" - because the v1 export endpoint was retired; instead link to /api/v2/exports
NOTE: this gate matches strings, not intent. A clean pass is not a judgement that the meaning is acceptable.
FAIL: 1 of 2 rules violated in site
```

The agent switches to `/api/v2/exports` and the gate passes. The reviewer still reads the release and finds “the old export URL keeps working for now” in an FAQ: no forbidden string, but it contradicts D-014. The sentence goes, and the review records why.

**Verdict:** gate Pass and review Pass for this release. No rule added: no exact string would have caught that sentence.

## Stop if

- The decision is disputed or reversed: update the decision record before touching the rule.
- The only way to pass is to weaken or delete a rule without its owner.
- The check exits 2: a missing or empty path, empty rules file or unreadable file is a failure, never a pass.
- A rule needs judgement (tone, intent, framing): send it to review instead of growing a pattern.
- Someone proposes `--no-verify` or skipping the CI job to ship.

## Sources

- Git: githooks — https://git-scm.com/docs/githooks (checked 2026-10-02)
- GNU: grep manual — https://www.gnu.org/software/grep/manual/grep.html (checked 2026-10-02)