Home / Research

Technical report

The limits of metadata-only detection on live npm malware

Sundar Shahi Thakuri · DepWall · 2026-09-20 · version 1.0

The benchmark that does not mean anything#

OSV carries a large and growing body of MAL-* advisories for npm — 221,357 entries in the dump used here. Running a detector over all of them and publishing a percentage is the obvious move, and it produces a number that cannot be interpreted.

Most packages in that corpus have been unpublished. Ask npm about one and you get a 404. Ask an install-time gate and you get BLOCK — on a signal whose message reads "not found in the registry". That verdict is correct, it protects nobody, and in a tally it is indistinguishable from having recognised the malware.

A detection rate computed that way measures npm's takedown queue wearing a detector's name. It goes up when the registry gets better, not when the tool does.

So the corpus is split by whether the flagged package is still installable, and only that subset is scored.

Method#

The npm advisory dump was fetched from OSV and every non-withdrawn MAL-* entry extracted. From 221,357 advisories a sample of 10,505 was drawn, stratified by publication year and weighted toward recent years, where surviving packages are concentrated. The sample is deterministic — fixed stride, no RNG — so two runs over the same dump examine the same packages.

Each sampled package was probed against the registry to establish whether the specific flagged versions still resolve. Survivors were then evaluated through npmEcosystem.check() — the production decision path — and the verdict recorded is the verdict a developer would have received.

The measurement ran on 2026-09-20 with the gray-zone LLM judge disabled. That is not incidental: these figures describe the metadata-only path — the one that runs on every package in every install — and a run with the judge switched on measures a different, larger system. This report refuses to build from a judge-enabled run rather than let the two be confused, because the first one we produced was judge-enabled and would have published that contradiction under this title.

Reachability, reported separately

The ratio between those rows is itself a finding, and it is why published detection rates against this corpus should be read with suspicion. The great majority of "malicious npm packages" in any advisory feed are historical.

Counting per package, not per version

One advisory routinely lists dozens of versions of a single package, and scoring all of them lets one name own the tally. Versions are capped at three per package; of 308 versions evaluated, 318 were dropped by that cap rather than silently truncated.

The headline is therefore per package — the unit a developer actually installs — taking each package's worst verdict across its evaluated versions.

The hash denylist is not what is being measured

DepWall ships an exact-match denylist of 4,947 malicious tarball integrity hashes, refreshed from https://osv-vulnerabilities.storage.googleapis.com/npm/all.zip. Exact-byte matching is strong — it cannot be evaded by renaming — and narrow: it recognises tarballs somebody has already reported. Only about 2% of advisories carry a hash at all. This report measures whether the inferential signals reach malware the denylist has not seen.

Results#

Stopped in total: 168/183 packages (91.8%). Allowed: 15 (8.2%). The ALLOW row is the result; everything else is context for it.

ASK deserves a note before it is read as a near-miss. An ASK stops the install and puts the package in front of a person; it does not pass it through. For a malicious package that is a materially different outcome from ALLOW. Counting the two together as "not blocked" understates the gate as much as counting ASK as a catch would overstate it, so they are reported separately.

Per version, underneath

The table above takes each package's worst verdict. Per evaluated version the split is BLOCK 129, ASK 149, ALLOW 30.

Those two views disagree by design, and the disagreement is a finding rather than an inconsistency. A package can have one version recognised and another not — an exact-hash denylist matches the tarballs somebody reported, and an attacker who publishes a fresh version publishes fresh bytes. Collapsing to the worst tier alone would hide exactly that limit.

The versions that were allowed

Published by name, because a miss described only as a percentage is not a miss anybody can check. These are versions, so a package listed here may still appear as BLOCK in the per-package table above on the strength of a different release:

• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• @spzhongwin/[email protected]
• @spzhongwin/[email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• @spx-smartsorting/[email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• @apiwizards/[email protected]
• [email protected]
• [email protected]

Why metadata cannot reach the allowed cases#

These are not tuning failures. They are packages whose malice is not expressed in any field a metadata check can read.

The payload is in the source, not the shape. A package with a plausible name, an ordinary publication history, no lifecycle scripts and a normal dependency list, whose exported function exfiltrates an environment variable when called, presents no anomalous metadata. There is nothing to read short of reading the code.

The name is not borrowed. Name-based signals need something to compare against. A malicious package under an original name is not near any popular name, is not in a hallucination feed, and is not a typo of anything.

The maturity signal decays as the package survives. Age and adoption flag a days-old package with no downloads. Malware that has sat in the registry for months accumulating incidental installs looks, to those fields, established. The signal is strongest exactly when the attack is newest and weakest on the long tail this corpus is mostly made of.

Provenance can be present and honest. npm provenance attests the build, not the source. A package built by a real pipeline from a repository whose contents are malicious carries a valid attestation.

What changed as a result#

Two things, and neither is a threshold adjustment. Lowering thresholds until this corpus scores well moves the cost onto the 3,213 popular packages measured in our false-positive baseline, where the interruption rate is currently 1.0%. A change that improves one of those numbers by spending the other is not an improvement.

The judge runs on the gray zone. Cases metadata cannot decide are escalated to content inspection rather than resolved by inference. If the evidence is in the source, the answer has to come from reading the source — and only where warranted, because sending every package's text to a model would be slow, costly and mostly pointless.

Every allowed case became a fixture. The packages listed above are in the red-team corpus, asserted at the verdict we actually produce — not the one we would prefer. A fixture asserting the desired answer is a wish; one asserting the real answer fails loudly the day behaviour drifts.

Limitations#

Reproduction#

git clone https://github.com/depwall/depwall-corpus
npm install && npm run build
node scripts/osv-bench.mjs --sample 400 --out data/osv-measurement.json
npm run gen-research

Every figure above is substituted from data/osv-measurement.json at build time; none is written into the source of this document. A figure the generator cannot derive withholds the report rather than publishing it with a gap.

Data availability

Benchmark harness, OSV hash denylist and verdict engine are in the DepWall repository; the measurement is reproducible with one command. https://github.com/depwall/depwall-corpus

Every figure in this report is substituted from the measurement output at build time rather than written into the text, so the document cannot carry a number that was not computed. The generator errors on an unresolved figure.

How to cite

@techreport{depwall2026the,
  title       = {The limits of metadata-only detection on live npm malware},
  author      = {{Sundar Shahi Thakuri} and {DepWall}},
  year        = {2026},
  institution = {DepWall},
  type        = {Technical report},
  url         = {https://depwall.com/research/the-limits-of-metadata-only-malware-detection/},
  note        = {Accessed: \today}
}