How I built a fail-closed vulnerability watcher
I wanted a vulnerability watcher that says when it does not know. In one seven-day run it read 11,080 advisories and promoted none for lab testing.
A fail-closed vulnerability watcher is a monitor that treats missing data as an unfinished result. If a feed fails, a package name only resembles an inventory entry or the installed version is unknown, the run is reported incomplete or the item stays on watch. It never marks that item “not affected”.
Fail open vs fail closed
When a control fails, it has two choices. Fail open lets the request through, which keeps the system available and leaves it unchecked. Fail closed blocks it, which keeps it checked and accepts the outage. An authorisation check usually fails closed; a powered lock on a fire exit fails open.
A watcher has nothing to block, so I apply the idea to its result. Missing data cannot turn into “not affected”, in the same way a failed check cannot turn into “allowed”.
A run can end “incomplete”
The rule is conservative by design. An item on watch stays there until the missing version arrives or the vendor advisory changes. The goal is a short list of decisions, each traceable to the advisories and inventory records behind it.
Why the response window is shrinking
On 24 July 2026 I wrote a short field note about a pattern my CVE work kept showing. AI is compressing the time between discovering a vulnerability, disclosing it publicly and weaponising it. The gap is shrinking from quarters towards hours. Four incidents show different parts of the same problem: SQLite’s CVE-2025-6965, Log4Shell, PAN-OS CVE-2024-3400 and the XZ Utils backdoor.
Their technical details differ, but each one asks the same operational question: how quickly can a team turn a weak signal into a decision someone owns? Once an advisory is public, every unverified dependency, exposed service and stale exception is on the clock.
My response is a faster intelligence loop. It combines continuous review, ranked patching, automated testing and monitoring that can run at machine speed, while I keep the consequential decisions. The automation shortens the delay between a signal, its priority and a verified change.
- Keep the inventory in a form I can query, so it never turns into a document that goes stale.
- Rank exposure by reachable impact and by what the inventory and advisories show. Headline severity is one input among several.
- Test the proposed fix, then check the live result after release.
- Record every unknown, so that “no alert” is never read as “safe”.
I wrote the July piece as a working field note, neither a vendor advisory nor an incident report. The longer argument is on LinkedIn.
The questions a monitor has to answer
A vulnerability list answers one question: what has been reported? A monitor I act on has to answer harder ones. Which records changed? Which ones map to software I run? Is the match exact? Is my version inside the affected range? Is exploitation known, predicted or merely possible? What is still missing?
Each public source contributes a different kind of fact. The CISA Known Exploited Vulnerabilities catalog lists vulnerabilities known to be exploited, and I use it to set priorities. The NVD vulnerability API supplies CVE records, CVSS metrics, references and applicability data. FIRST EPSS estimates the probability of exploitation. The GitHub Advisory Database adds ecosystem and package context.
KEV, EPSS and CVSS each answer a different question. None of these sources can prove on its own that one of my systems is exposed. That takes my local inventory and a recorded installed version for each product.
The six-stage pipeline
I designed the watcher as a sequence of gates. Each stage narrows the question, and each move from one stage to the next leaves a record. An item that fails a gate keeps the reason it failed.
- CollectFetch the authoritative feeds and store each one’s health, retrieval time and raw response.
- NormaliseMap each source’s schema onto one advisory record and keep where every field came from.
- MatchCompare vendor, product, package and ecosystem with my maintained inventory.
- EnrichAdd exploitation status, exploit probability, severity, affected ranges and vendor context.
- GateRequire a recorded installed version and one defined defensive objective.
- RecordWrite WATCH, REVIEW, rejected or incomplete, and keep the reason.
The monitor has four possible outcomes: WATCH, REVIEW FOR LAB, INCOMPLETE and NOT APPLICABLE.
Normalisation keeps what a later review needs. For every advisory I store the source, retrieval time, advisory identifiers, affected ranges and enough of the raw response to show how a decision was reached. I keep the stored response because it lets me reproduce the decision.
Feed health is part of the result
The easiest dangerous mistake is turning a network or parser failure into zero findings. If a required source returns nothing because DNS failed, an API changed or the runtime had no network access, that zero says nothing about security. It is an observer failure, and the run is marked incomplete.
- Observed
- A fresh response, checked and stored.
- Empty
- The query worked and returned zero results.
- Blocked
- Something in the environment stopped a required check.
- Unknown
- No usable data yet: missing, stale, ambiguous or not collected.
The run record and the summary I read keep those four states apart. When a feed fails, I keep the failed run, mark the catalogue incomplete and retry the same collection step once the environment allows it. A failed run is never filed as a clean scan.
Product identity comes before urgency
Similar names create false confidence. One recent match tied CVE-2021-21315 to a Node.js entry in my inventory, because the advisory described a “System Information Library for Node.JS.” The affected component is the systeminformation npm package. The Node.js runtime only shares a word with the advisory’s description.
The same failure appears when an advisory for an integration package is mistaken for the application it connects to. “n8n-MCP” and “n8n” are separate identities in my inventory. A high severity score does not make a wrong match right.
An agent helps most here as a sceptical reconciler. It compares aliases, source descriptions, package ecosystems and earlier decisions. The deterministic gate still decides whether vendor, product, ecosystem and package identifier agree.
The installed-version gate
Once the product matches, two facts matter more than severity. Which version is installed? Does an authoritative advisory place that version inside the affected range?
The promotion gate needs all five inputs present: the exact product identity, the affected and fixed range, the installed version, a primary advisory and a safe detection objective. An item leaves it with one of three outcomes.
- WATCH
- Relevant signal; exposure not established.
- REVIEW
- Exact product and version support a scoped lab test.
- REJECT
- Wrong product, version or scope.
A missing installed version blocks the key inference, so the item cannot be promoted. The watcher keeps it visible and tells me which installed version to collect next.
Public systems stay passive; active work belongs in the lab
The monitor reads metadata only. It never downloads exploit code, scans public targets or tries to confirm a vulnerability on production. Those actions need different permissions and different safety controls.
An item moves towards active validation only when five conditions hold. The target is an authorised, disposable replica; the product and version are exact; the detection objective is defined; logging and containment are ready; and the reset has been proven. The test asks whether a defensive control sees one defined scenario on a disposable target.
- Build the target from known inputs. Pin the vulnerable component and capture its checksum or image digest.
- Define one detection objective. Name the event, log, alert or control expected to fire.
- Contain every route. Keep the target isolated from public and production networks.
- Keep the lab notes. Save the scenario, observations, timestamps and limitations.
- Reset and prove it. Destroy or restore the target, then verify the baseline.
Passive monitoring can raise a question, and it never authorises active testing by itself. Promotion changes the kind of work, so it needs an exact product and version match and my explicit approval.
What one seven-day run showed
This snapshot covers seven days and was the latest when I published this note on 30 July 2026. The right result was to promote nothing. Six feed and validation lanes completed. The watcher normalised 11,080 advisories and found 61 inventory-relevant signals. After identity checks, history reconciliation and the gates, 60 stayed on watch and zero were promoted for lab review.
| Stage | Result | What it means |
|---|---|---|
| Feed health | 6 of 6 complete | The observation window was usable; no required lane failed. |
| Normalised advisories | 11,080 | Every advisory read from the feeds. On their own they establish no exposure. |
| Inventory-relevant signals | 61 | Advisories worth checking against the products I run. |
| WATCH | 60 | Relevant enough to keep; exposure unproven. |
| REVIEW FOR LAB | 0 | No item passed every promotion gate. |
| Missing installed versions | 8 assets | The next task is recording those versions. Active testing waits. |
Seven apparent alert changes turned out to be duplicates or immaterial differences when compared with the previous successful catalogue. Because the decision history was kept, the watcher did not interrupt me again with conclusions I had already seen.
Questions I would ask first
What does fail-closed mean in vulnerability monitoring?
The system may reach a reassuring conclusion only when every required input is present. A failed feed, an ambiguous match or an unknown installed version leaves the item incomplete or on watch.
Is a CISA KEV or high EPSS score proof that my system is vulnerable?
No. KEV indicates known exploitation of that vulnerability, and EPSS estimates exploitation probability. Local exposure still depends on exact product identity, affected version, installed version, configuration and reachability.
Why keep WATCH items if they are not confirmed exposure?
They keep relevant signals and the missing facts in view without overstating certainty. A watch item can become actionable when a vendor advisory changes or the missing installed version arrives.
Should the watcher automatically run exploits?
No. Public and production systems remain passive. Any active validation belongs on an authorised, isolated, disposable target with a defined defensive objective, logging, containment and reset.
Sources
- CISA Known Exploited Vulnerabilities Catalog ↗: confirmed exploitation signal and an input for remediation priority.
- NVD Vulnerability API ↗: CVE records, applicability, metrics, references and change history.
- FIRST Exploit Prediction Scoring System ↗: an estimate of exploitation probability.
- GitHub Advisory Database documentation ↗: advisory search by ecosystem and package.
- Ubuntu CVE reports ↗: package status and remediation context specific to the distribution.
← Previous: Codex on my phone: steering agent tasks while the Mac works
Want this inside your team’s workflow? The Verified AI Operations Sprint tests one workflow in ten business days. See the sprint →