DepWallZero false positives is also what a broken scan reports
flagged: 0 is both the result the measurement exists to produce and exactly what a silently muted rule prints. So the harness now re-proves, from inert fixtures, that the scan it just ran was capable of failing.
Every false-positive number we published for the last three detections — the npm
require hop, the directory-main redirect, the pip import hop — came from
throwaway scripts in a scratchpad nobody can re-run. The numbers were real when they were
taken. They were also unreproducible the following week, which makes them assertions rather
than measurements.
The harness that should have owned those numbers never touched the code path they live
in. Phase B ran with --network none over pre-unpacked directories, so it never
saw a tarball fetch, an entry-point selection, or a follow round — precisely the three
things the new detections consist of.
Measuring the path production actually runs#
Phase A now also downloads real npm tarballs and PyPI sdists of the most-downloaded
packages — 150 each by default — capped at the same MAX_COMPRESSED_BYTES
production refuses to exceed. Using a different cap in the harness would measure a walker
nobody ships.
Phase B reads them off disk and hands the bytes to the production
fetchArchiveFiles through a local fetchImpl. The real walker, its
real caps and its real follow rounds all run, with the network still off.
Nothing is unpacked and no package code executes. The tar is walked in memory, exactly as the product does it. That constraint is not incidental — a false-positive sweep that detonates 300 real packages to measure them is a worse idea than not measuring them.
An honest denominator#
npm archives with no lifecycle script, and sdists with no setup.py, are
counted but not scanned — because production never fetches them either. They stay in the
denominator as what they are: packages this code path does not apply to. Dropping them
would inflate the coverage figure; scanning them would inflate the safety one.
Flagged benign packages are named, not merely counted. A false positive is a real developer halted at a real install, and the next person reading the summary needs to know which package it was, not that there was one.
The control, and why a zero needs one#
Here is the problem this section exists for. flagged: 0 is the result the
measurement is hoping for. It is also exactly what a silently broken scan prints.
Muted rules. An unwired follow round. Resolution returning nothing. Every one of those looks identical to a clean bill of health, and every one of them is a plausible thing to introduce while refactoring the file the rules live in.
So the run now re-proves, from the repository's own inert fixtures, that the scan it just performed was capable of failing:
- Resolution still names the deferred file.
- The rules still fire on the payload, and still stay quiet on the decoy.
- The follow rounds actually ran over the corpus.
On failure the run says so loudly and summary.json carries
bodyScanControl.ok=false. The zero is still printed — and it means
nothing, in machine-readable form, so that a downstream reader cannot quote it by
accident.
Proving the zero is a measurement#
Verified end to end, outside docker, against real archives — esbuild, core-js, sharp, lodash, nodeenv, six, dill. Result: 0 flagged, control green.
Then the other direction, which is the one that makes the first number mean something.
Planting inert known-bad archives, built from the repo's own fixtures, flags both
ecosystems and names the followed file: lib/setup.js for npm,
main.py for PyPI. The scan can produce a non-zero. Therefore its zero is a
measurement rather than an artifact of a scan that never ran.
The control had a bug, which is the best possible outcome#
The control's own failure path was tested too. It had a bug: an early return omitted
failed, so a run with missing fixtures crashed the harness instead of warning
about it.
Worth sitting with. The mechanism whose entire job is to make a broken run distinguishable from a clean one was itself broken on one path, and it was found only by deliberately breaking it. Untested error handling is not error handling — it is a comment about intent that happens to compile.
What this still does not tell you#
150 of the most-downloaded packages per ecosystem is a sample of the head of the distribution, and the head is where the well-maintained code lives. A false-positive rate measured there is a lower bound on the rate a developer with an ordinary dependency tree would see.
What changed is not that the number is now correct in some absolute sense. It is that the number is now produced by a command anyone can re-run, over the code path it claims to describe, with a control that fails when the measurement stops working.