# Provenance ledger

Version: 1.0
Updated: 2026-10-02
Method: https://suhailmohebi.com/notes/provenance-before-conclusions/

Purpose: let every conclusion travel backwards—from claim, to transformation, to the exact bytes it came from.

Output: an archive with untouched raw files, a hashed manifest, a claim ledger, a rerun recipe and a written rejection test.

## Use when

You are about to analyse downloaded or agent-collected data, and someone will later ask where a number came from, whether it still holds, or why the work stopped.

## Quick start

1. Create the layout in section 1; fetch every source with the section 3 script so each file gets its manifest line on arrival.
2. Record failures, blocks and empty responses as observations, never as “no data”.
3. Write each claim with its files, derivation, period and caveat, plus the result that would make you drop it.

A download is not evidence until the request, response, file, hash and limitation travel together.

## 1. Archive layout

```text
archive/
  raw/           original bytes, named by their SHA-256; never edited, never overwritten
  derived/       tables and profiles rebuilt from raw/ by a recorded command
  claims/        the claim ledger and its rejection tests
  manifest.tsv   one line per fetch, written during acquisition
```

- Re-fetching creates a new manifest line, not a replacement.
- Every derived file records its input hashes, the command and revision that built it, and what it excluded. If no recorded command can rebuild it, anything built on it is untraceable: rebuild it or drop it.

## 2. Source intake

Apply SIFT before a source enters `raw/`:

- [ ] **Stop.** Know what the source is before you read its data.
- [ ] **Investigate the source.** Who publishes it, under which licence or terms, and with what interest?
- [ ] **Find better coverage.** Is there a more original or more authoritative publisher of the same data?
- [ ] **Trace claims to the original context.** If it cites another dataset, fetch that one too.

## 3. Fetch with a receipt

Needs `curl` and `sha256sum` or `shasum -a 256`. Writes only inside the archive; exits with curl’s exit code after writing the receipt.

```bash
#!/usr/bin/env bash
# fetch-raw.sh - save one URL's bytes in raw/ and append a receipt to manifest.tsv.
# Usage: bash fetch-raw.sh URL [ARCHIVE_DIR]   (default: archive; writes only inside it)
set -u
url=${1:?usage: fetch-raw.sh URL [ARCHIVE_DIR]}
dir=${2:-archive}
mkdir -p "$dir/raw" || exit 2
manifest="$dir/manifest.tsv"
[ -f "$manifest" ] ||
  printf 'retrieved_utc\turl\tstatus\tfinal_url\tcontent_type\tbytes\tsha256\tcurl_exit\tmethod\n' > "$manifest"

sha256() { if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1"; else shasum -a 256 "$1"; fi | cut -d' ' -f1; }
method="$(curl -V | head -n 1 | cut -d' ' -f1-2) -sSL --max-time 60"
when=$(date -u +%Y-%m-%dT%H:%M:%SZ)
tmp="$dir/raw/.tmp.$$"
meta=$(curl -sSL --max-time 60 -o "$tmp" -w '%{http_code}\t%{url_effective}\t%{content_type}' "$url")
rc=$?
[ -f "$tmp" ] || : > "$tmp"    # a failed request is still an observation
bytes=$(wc -c < "$tmp" | tr -d ' ')
sum=$(sha256 "$tmp")
if [ -e "$dir/raw/$sum" ]; then rm -f "$tmp"; else mv "$tmp" "$dir/raw/$sum"; fi    # never overwrite raw bytes
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$when" "$url" "$meta" "$bytes" "$sum" "$rc" "$method" >> "$manifest"
printf 'status=%s bytes=%s curl_exit=%s file=raw/%s\n' "$(printf '%s\n' "$meta" | cut -f1)" "$bytes" "$rc" "$sum"
exit "$rc"
```

Error pages are kept as evidence; a 404 or 429 still exits 0. A failure before any bytes arrive is status `000` with zero bytes. A 200 with a non-zero `curl_exit` holds partial bytes (28: `--max-time 60` cut the download): classify it Blocked.

## 4. Manifest

`manifest.tsv` is the machine receipt (time, URL, status, media type, bytes, hash, curl exit, method). Add one row per raw file for what a machine cannot know; an edited raw file is detectable (section 6).

| File (raw/SHA-256) | Owner and dataset | Licence or terms | State and status | Rows after validation | Limitations and refresh policy |
|---|---|---|---|---|---|
|  |  |  |  |  |  |

Negative observations are data. Classify every fetch:

| State | Meaning | Record |
|---|---|---|
| Observed | valid response with records | row count after profiling |
| Empty | valid response with zero records | the query and the empty result |
| Blocked | access path unavailable: error status, refusal, timeout, or any non-zero curl exit | status or curl exit code, retry plan |
| Unknown | not yet established, such as a 200 with zero bytes | the check that would establish it |

## 5. Claim ledger

Write each claim as: “Between [start] and [end], [measure] for [sample and filters] was [value], by [derivation]; caveat: [limit].”

| Claim | Supporting files | Derivation | Confidence | Counter-evidence and negative observations | Status |
|---|---|---|---|---|---|
| C1: |  |  | High / Medium / Low |  | Supported / Narrowed / Dropped |

## 6. Rerun recipe

Write this before the first run; run it beside `archive/` and `fetch-raw.sh`. A rerun works offline from `raw/`; a refresh is a separate decision.

```bash
sha256() { if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1"; else shasum -a 256 "$1"; fi | cut -d' ' -f1; }
# 1. raw/ is intact: names equal hashes; raw files and manifest lines match one to one.
for f in archive/raw/*; do [ -e "$f" ] || continue; h=${f##*/}
  [ "$(sha256 "$f")" = "$h" ] || echo "EDITED RAW FILE: $f"
  cut -f7 archive/manifest.tsv | grep -qxF "$h" || echo "NO MANIFEST LINE: $f"; done
tail -n +2 archive/manifest.tsv | cut -f7 | sort -u | while IFS= read -r h; do [ -e "archive/raw/$h" ] || echo "MISSING RAW FILE: $h"; done
# 2. Refresh only when a source or decision needs it: re-fetch every source as a new observation.
# tail -n +2 archive/manifest.tsv | cut -f2 | sort -u | while IFS= read -r u; do bash fetch-raw.sh "$u" archive; done
# 3. After a refresh: sources whose latest bytes differ from the previous fetch.
awk -F'\t' 'NR > 1 { prev[$2] = last[$2]; last[$2] = $7 }
  END { for (u in last) if (prev[u] != "" && prev[u] != last[u]) print "CHANGED: " u }' archive/manifest.tsv
```

Any line from step 1 stops the rerun. Otherwise rebuild `derived/` from the input hashes it records, compare row counts and hashes with the last run, and re-run every rejection test.

## 7. Rejection test

> Drop [claim] if [observation] appears when [derivation] is re-run on [source or date].

> Close the project if [the joins are too weak / the licence does not allow the use / the data supports cohorts but not individual predictions / the product already exists].

- Decision: Go / Narrow / No-go
- Evidence (manifest rows and derived files):
- Decided by and date:

## Worked example

Question: did permit approvals get faster in 2026? Three fetches on 2026-10-02:

| File (raw/SHA-256) | Owner and dataset | Licence or terms | State and status | Rows after validation | Limitations and refresh policy |
|---|---|---|---|---|---|
| raw/4c1d… | data.example.org: residential permits 2025 | open, attribution | Observed: 200 | 4,212; 14 lack a decision date | residential only; refresh quarterly |
| raw/e3b0… | data.example.org: residential permits 2026 | open, attribution | Unknown: 200 with 0 bytes | none | retry after the next portal refresh |
| raw/9a77… | data.example.org: permits API, page 1 | terms unclear | Blocked: 429 refusal | none | unusable until the terms are known |

- **C1:** In 2025, residential permits in `raw/4c1d…` took a median of 34 days from application to approval (n = 4,212 − 14 undecided = 4,198; `permits.py` at `abc1234`); caveat: one portal’s residential-only export, undecided cases excluded. Supported, high confidence. Drop if a refresh moves the median by more than two days.
- **C2:** Approvals got faster in 2026. Dropped: a zero-byte export and a refused API call are neither zero permits nor a trend.

**Decision:** Narrow. Publish C1 with its period and caveat; record C2 as not established; retry the 2026 export after the next portal refresh.

## Stop if

- A claim rests on a file with no manifest line, or on a raw file whose hash no longer matches its name.
- A derived table cannot be rebuilt from `raw/` with the recorded command.
- A claim depends on a source whose licence or terms are unknown or do not allow the intended use.
- An access failure or an empty body is about to be reported as an absence (“no data”, “zero”).
- The rejection test fired: narrow or drop the claim instead of rewording it until it passes.

## Sources

- Mike Caulfield (Hapgood): SIFT (The Four Moves) — https://hapgood.us/2019/06/19/sift-the-four-moves/ (checked 2026-10-02)