ADR-0031: Native Dependency, Secret and Misconfiguration Scanner

• Status: Accepted. The five scanners, the orchestrator, the CLI verb and the Layer-1.5 hook wiring are implemented and run in CI. The advisory snapshot is generated at release time and is not committed to the repository.
• Date: 2026-08-03
• Supersedes: None
• Related: ADR-0020 (the layer this extends), ADR-0019 (the same prose-to-gate move on the plan side), ADR-0002, ADR-0005, ADR-0001

Context and Problem Statement

ADR-0020 gave the in-session review one deterministic layer: sixteen regex rules in lib/security/patterns.cjs. That layer knew nothing about CVEs, nothing about infrastructure misconfiguration, nothing about licences, and carried four secret rules.

At the same time skills/np-dependency-audit/SKILL.md demanded, in its own verification bar, "free of known critical CVEs in the pinned version (run the ecosystem's audit/advisory check)" and "Exact name, no typosquat". Nothing in the repository ran either check. The bar was prose, and an executor could satisfy it by asserting it.

That is the same shape ADR-0019 resolved on the plan side: a rule enforced by an agent reading English is a rule that holds until it is inconvenient.

Decision Drivers

• A checkable claim beats a stated one. The point is not more findings; it is that the dependency bar stops being self-reported.
• [ADR-0002] Zero runtime dependencies. No scanner binary, no SDK, no vulnerability-database client.
• [ADR-0001] No daemon. Every scan is a fresh, short-lived node process, so load cost dominates match cost.
• [ADR-0020] Never block. A scanner wired into a write hook that can abort a write is worse than no scanner.
• A false positive is a switched-off feature. Noise on ordinary repository content is the failure mode that ends the feature, not a missing rule.

Considered Options

• A: Shell out to an external scanner binary. Rejected: an external binary is an environment assumption we cannot rely on, and it would carry a several-hundred-megabyte vulnerability database per machine.
• B: Build the deterministic scanners, skip the vulnerability half. Rejected: the CVE bar is precisely the one the skill already demanded and nothing enforced.
• C: Build all five, source the advisory data. Chosen.
• D: Expose the scan over a protocol server. Rejected — see below.

Decision Outcome

Chosen: five native scanners under lib/scan/, one finding shape, and a shipped advisory snapshot.

Scanner	Rules	Nature
Secret detection	95 (NPS-0100..0194)	provider-shape regexes plus a charset-aware entropy promoter
Misconfiguration	52 (NPS-0500..0729)	CI workflows, Kubernetes, Compose, container buildfiles, HCL
Vulnerability match	NPS-0300..0303	version-range matching over a shipped advisory snapshot
Malicious package	NPS-0400	sorted-name binary search over the malicious-package records
Licence policy	NPS-0801..0806	SPDX classification over the dependency inventory

The dependency inventory is the foundation the last three join on: eight ecosystems, fourteen manifest and lockfile formats, normalised to one package record carrying a package URL, a resolved scope and its source file.

The data is sourced, not invented

A vulnerability corpus is a dataset, not an algorithm. The matcher is built; the advisories are shipped.

scripts/build-advisory-db.cjs runs at release time: it fetches one archive per language ecosystem over anonymous HTTPS, filters by a source-licence allowlist (lib/scan/../scripts/advisory-compact.cjs), discards withdrawn records, drops the fields the match does not need, and writes per-ecosystem gzip shards plus a manifest.json carrying generated_at, per-ecosystem record counts, source licences and a SHA-256 per shard.

Measured, not estimated: the compaction achieves 89× against the raw input, at ≈16 bytes per range advisory and ≈8 bytes per malicious-package record after gzip. Extrapolated to the real corpus that is ≈2.3 MB. The synthetic corpus used for the measurement is more repetitive than the real one, so treat 2.3 MB as a floor and the 4–6 MB budget as the working ceiling; a build exceeding 8 MB reduces the malicious list to the top ecosystems first.

Dropping the long details prose does most of that work.

Where the snapshot lives — and where it does not

ADR-0002 and ADR-0005 both pin "the install-payload tree contains only .cjs files and Markdown." The snapshot therefore never enters the payload.

It ships inside the npm package under lib/scan/data/ and is copied once at install time into ~/.nubos-pilot/advisory-db/<version>/. Ten projects on one machine share one copy, both ADR invariants stand, and a scan needs no network.

Network exists at exactly one point in the lifecycle: the release-time build.

Sharded and lazily loaded, because there is no daemon

ADR-0001 means every scan pays start-up cost. So the store is sharded per ecosystem and keyed by normalised package name, and only the shards the inventory references are opened — a Node-only project never touches the Python shard, and a test asserts that. The malicious list stays a sorted newline-delimited buffer searched by bisection rather than a parsed structure: 17 probes over 100 000 records, against ~50 000 for a linear scan, with no per-process Set construction.

Refusing to answer is a first-class result

Every path that cannot decide says so instead of returning nothing:

• no snapshot → NPS-0301
• snapshot fails its hash → NPS-0302, and no vulnerability verdict is issued from that shard
• an ecosystem with no version comparator → NPS-0303
• a Terraform value that is a variable or a function call → NPS-0679, never a confident pass
• a Go module replaced by a local path → the upstream version is dropped, because claiming it would match advisories against code that is not in the build

A tampered advisory store produces silently false negatives, which is strictly worse than no store. Hence the error-severity doctor check and the digest verification before decompression.

Version comparison is the real risk, and it is differentially tested

A wrong range comparison is a silent false negative — the worst failure class here. lib/scan/advisory/ranges.cjs implements eight ecosystems, four of which do not follow SemVer, and each comparator was differentially tested against the ecosystem's own reference implementation: over 320 000 version pairs, zero mismatches.

That found two real false-negative generators that specification-derived tests had missed:

1. Java — 5.2.25.RELEASE and 4.1.0.Final must compare equal to 5.2.25 and 4.1.0. Spring and Netty artifacts use exactly those suffixes, so the naive port would have reported every one of them as unaffected.
2. PHP — 1.0.0-stable1 must drop the stability modifier and equal 1.0.0.

An unparseable installed version is skipped; an unparseable advisory boundary throws. The asymmetry is deliberate: bad data in the shipped store must be loud, while a package we cannot parse must not abort the scan.

On a protocol server

Rejected, on the same grounds ADR-0030 reasons from. A protocol tool can only be invoked by the model, and Layer 1 exists precisely because it runs without one. Exposing the scan that way would trade a free deterministic layer for a model-gated one, add a long-lived process against ADR-0001, and pull in the SDK ADR-0002 already refused. Agents reach the scanner the way they reach everything else: node np-tools.cjs scan ….

Consequences

Good, because:

• The dependency-audit bar is now run rather than asserted, and the skill names the command per bar.
• NPS-0012, the blanket "review this workflow file" reminder, is retired — replaced by eight real checks including the pull_request_target plus untrusted-checkout takeover primitive at critical.
• Zero model cost. The whole deterministic layer runs in a hook without a token.
• The malicious-package records mechanise the typosquat bar for free — a set lookup, no range logic.

Bad, because:

• The snapshot is frozen at release. scan db-update fetches a delta and scan db-status reports the age, but a CVE published the day after a release is invisible until the next one. Every finding set carries db_generated_at so the staleness is visible rather than silent.
• The match is naive next to a mature scanner. There is no reachability analysis and no vendor suppression data. Mitigated by excluding dev-scope packages from gating by default; without that the ledger floods and the feature gets switched off.
• Maintenance: 147 rules and an HCL subset parser are a standing cost. The HCL parser recognises and skips what it cannot evaluate rather than guessing, which bounds the damage but not the upkeep.

Deliberately out of scope

Container and OS-image scanning, VM images, live-cluster scanning, and distribution advisories. Those need a registry client and parsers for binary OS package databases, and the distribution feeds are the ones the licence allowlist excludes. A dev-time planning tool does not scan operating systems. This ADR covers source repositories the agent is writing, and says so rather than implying parity with a full scanner.

More Information

• Library: lib/scan/ — finding.cjs (one shape, five-step severity, NPS-#### ids), walk.cjs (one file pass, many extractors), inventory/, advisory/, secrets/, misconfig/, license/, sbom/.
• Release tooling: scripts/build-advisory-db.cjs, scripts/advisory-compact.cjs, scripts/cvss.cjs, scripts/zipread.cjs — authoring-time only, never shipped to a user.
• CLI: np-tools scan all|inventory|advisory|secrets|misconfig|license|sbom|db-status, --json, --fail-on.
• Hook wiring: Layer 1.5 inside the existing security scan verb. The hook script and its five registrations are unchanged.
• Config: security.scan in config.json. fail_on defaults to never — ADR-0020 §"Non-blocking by construction" is not weakened by this ADR.
• Attribution: the advisory sources are named in ATTRIBUTIONS.md. The attribution-only licences require it; that file is the one place those names belong.