Home / Research

Technical report

What an install-time gate costs on packages nobody should stop

Sundar Shahi Thakuri · DepWall · 2026-09-20 · version 1.0

Why measure the quiet case#

A security gate has two error modes and the literature reports one of them. Papers count what a detector catches; almost nothing counts what the same detector does to the packages that were never an attack.

For an install-time gate that asymmetry is backwards, because the second number is the one that determines whether the tool survives contact with a working developer. A gate that interrupts one install in twenty is a gate that gets uninstalled in a week, and a gate nobody runs has a detection rate of zero regardless of what it scored on a corpus.

Our own published miss rate — 69% of still-installable OSV-flagged malware allowed on metadata alone — is only half a picture. This report is the other half.

Method#

The corpus is drawn from src/data/popular-names.json, the set DepWall already carries as its popularity dictionary for edit-distance comparison. It is a reasonable stand-in for "packages a developer is plausibly about to install", and it has the useful property of being the same list the slopsquat signal compares against — so any name in it is, by construction, a name the tool considers established.

Scoped packages are excluded, and the exclusion is large enough to state up front: 2,759 of 5,973 names (46.2%) begin with @ and are not measured here, leaving 3,214 attempted.

The reason is mechanical rather than principled. npm's bulk downloads endpoint does not accept scoped names, and per-package lookups for them resolve about 9.6% of the time at this volume — measured at 169 of 1,750, and unchanged by pacing or concurrency. As the next section explains, an unresolved download count does not degrade gracefully here: it becomes an ASK. Including scoped packages would therefore have meant publishing thousands of interruptions that describe npm's API rather than DepWall. A narrower corpus that is right beats a wider one that is quietly wrong, so the scoped half is simply not claimed. Anything below about npm-wide behaviour should be read as being about its unscoped majority.

For each name we fetched live registry metadata for the latest dist-tag via the same fetchPackageRecord() the CLI uses, then called evaluate() — the production decision path — on the resulting record. No separate scoring system was built for this report. The verdict recorded is the verdict a developer would have received.

The measurement ran on 2026-09-20. Of 3,214 names attempted, 0 failed to resolve against the registry and 1 returned 404, leaving 3,213 packages in the analysed set. Unresolved names are excluded rather than counted as clean: a package we could not reach is unknown, and this report does not convert unknown into a passing grade.

Why the download count decides the result

One methodological detail dominates this measurement, and it is worth stating before the numbers rather than in a footnote after them.

DepWall's maturity signal returns warn — an ASK — when a package's age or weekly download count cannot be verified. That is the right behaviour at an install gate: an unverifiable package is unknown, and unknown never resolves to a silent allow. It also means a measurement harness that gets rate-limited by npm's downloads API produces a record indistinguishable from a package with no adoption.

Our first run made this concrete. Fetching download counts one package at a time was throttled, the counts came back null, and the run reported an interruption rate of 96.7% on the most-depended-on packages on npm. That figure described our own request pacing and nothing about the gate. The harness now takes download counts from npm's bulk endpoint, and a package whose count still cannot be resolved is recorded as an error and excluded rather than counted as a quiet ASK.

Download counts were resolved for 100.0% of the analysed set (3,213 of 3,213). This report refuses to build below 95%, because underneath that the headline stops being a property of DepWall.

What this method deliberately does not do

It does not download tarballs, read source, or invoke the LLM judge. It measures the metadata-only decision path — the one that runs on every package in every install, and therefore the one whose false-positive rate a developer actually feels. The judge runs only in the gray zone and is out of scope here.

Results#

The headline figure is the interruption rate: 32 of 3,213 packages (1.0%) returned something other than ALLOW. That is the proportion of established, widely-depended-on packages on which this gate would put itself in a developer's way.

Which signals fired

4 distinct signals fired at least once across the corpus:

Severity is what separates a note from an interruption. Only critical findings produce BLOCK and only warn produces ASK; info findings are recorded on the verdict and shown in output, but they do not stop anything. A signal with a large count and info severity is not a false-positive problem — it is an observation the tool is making without acting on it.

Structural facts about the corpus

Two of these are worth recording independently of the verdicts, because they are measurements of npm rather than of DepWall:

The attestation figure is the one with consequences beyond this report. npm provenance is the strongest available signal for account takeover — a release published without the pipeline every previous release used is a genuine discontinuity. Its usefulness is bounded by adoption, and adoption across the most-depended-on packages on the registry is what the second row measures.

The install-script figure is the counterweight to a common recommendation. Published analyses of malicious packages put lifecycle-script abuse at 72.21% of samples in one benchmark of 6,420 packages, and 46% in JFrog's reporting over a comparable window. If a comparable share of legitimate popular packages also runs install scripts, then "has a postinstall" cannot function as a verdict on its own — which is why DepWall records it as evidence and weighs it rather than blocking on it.

Interpretation#

We separate the non-ALLOW results into three kinds, because they carry different obligations.

Defensible interruptions. A package that genuinely resolves a dependency to a git URL, or has genuinely lost attestation it previously carried, is a package the gate is correct to raise. The developer is inconvenienced and the tool is right. These are a cost, not a bug.

Signal-design defects. A finding that fires on a structural property shared by a large fraction of healthy packages is not describing risk, it is describing npm. Any signal appearing at high volume in the table above against a corpus of established packages is on this list by definition and gets re-examined against the corpus before it is allowed to keep its severity.

Corpus artefacts. The popularity list is a snapshot. Packages in it get deprecated, transferred and unpublished, and a verdict driven by that is measuring list staleness rather than detection quality — which is why the deprecated and 404 counts are reported separately above rather than folded into the headline.

Limitations#

Reproduction#

git clone https://github.com/depwall/depwall-corpus
npm install && npm run fetch-package-data
npm run gen-research

The fetch writes one JSON record per package to data/npm-cache.jsonl, including the full finding list. Every figure in this report is substituted from that file when the page is generated; none is written into the source text. A figure the generator cannot derive is a build error rather than a silent omission, which is the mechanism that keeps this document honest as the data moves underneath it.

Numbers will differ from ours on any later run, because the registry moves. That is the point of publishing the harness rather than only the table.

Data availability

Package list, fetch harness and verdict engine are in the DepWall repository; the measurement is reproducible with two commands. https://github.com/depwall/depwall-corpus

Every figure in this report is substituted from the measurement output at build time rather than written into the text, so the document cannot carry a number that was not computed. The generator errors on an unresolved figure.

How to cite

@techreport{depwall2026what,
  title       = {What an install-time gate costs on packages nobody should stop},
  author      = {{Sundar Shahi Thakuri} and {DepWall}},
  year        = {2026},
  institution = {DepWall},
  type        = {Technical report},
  url         = {https://depwall.com/research/what-an-install-gate-costs-on-popular-packages/},
  note        = {Accessed: \today}
}