DepWall69% of live npm malware walked past our metadata checks
We ran our own engine over the npm packages OSV flags malicious that are still installable today. Three blocked, eight asked, twenty-five were allowed. Here is why, and what we changed.
Every install-time check we had for npm reads metadata and infers from it. Does the package run an install script. Is the name one edit away from something popular. Is it too new, does anyone download it. That is all a gate can see before install, and for a long time we assumed it was mostly enough.
It is not, and we can now say by how much.
The measurement#
OSV carries about 219,000 MAL-* advisories for npm — packages somebody has
confirmed malicious. The obvious benchmark is to run all of them through the gate and
publish a percentage. That benchmark is worthless, and it is worth saying why before the
number that matters.
Almost every package in that corpus has been unpublished. Ask npm about one and you get a 404. Ask DepWall and you get BLOCK — on a signal whose message reads "not found in the registry". True, useless, and indistinguishable in a tally from having recognised the malware. A headline built on it would be measuring npm's takedown queue and calling it detection.
So we split the corpus by whether the malicious version is still installable, and counted only those. Those are the ones a developer could still pull today, and the only ones where a verdict is a claim about the package rather than about its absence.
| Verdict | Versions |
|---|---|
| BLOCK | 3 |
| ASK | 8 |
| ALLOW | 25 |
Thirty-six installable versions across nineteen packages. Sixty-nine percent walked straight through.
Why — and it is structural, not a tuning problem#
We read the ones that got through. Two shapes account for almost all of them, and neither is something a metadata check can ever see.
The payload is a prebuilt binary
@ccfly/setup-darwin-x64 ships a compiled darwin/x64 Mach-O binary containing a
Go agent that dials a hardcoded broker over WebSocket. There is no lifecycle script,
because there is nothing to script — the artifact is the payload. The manifest is
unremarkable. The name resembles nothing.
The payload runs on require()
One package's index.js loads a helper that picks a platform-specific endpoint and
downloads an opaque executable from hosts assembled by string concatenation. None of that
happens at install. It happens the first time your application imports the package, which
is after install and after every metadata check has already passed.
From metadata, both of these are indistinguishable from a small, boring, legitimate package. They should be — from metadata they are identical.
What we changed#
npm publishes a dist.integrity for every version: the SHA-512 of that exact
tarball. OSV publishes the same digest for a few thousand of the packages it has
confirmed malicious. Comparing the two is not a judgement call — the registry is serving
those bytes or it is not — so it is the one signal we have with no false-positive rate to
trade off, and the only one allowed to BLOCK on its own.
On a fresh sample, every live malicious package is now gated, and six of ten are gated by that signal alone. Strip it and they return to ALLOW, because nothing else has anything to say about them.
What it still does not do#
We would rather publish this than have you discover it:
- Coverage is about 2%. Only around 4,000 of OSV's ~219,000 malicious npm advisories carry a digest at all.
- It pins one version. An advisory routinely names ten malicious versions and supplies a digest for one.
- Republishing defeats it completely. Change one byte, get a new digest, no match. That is not a weakness to be fixed — it is what "exact" means, and it is the same property that buys the zero false-positive rate.
The heuristics still have to catch the recompiled version. That is why the digest list is additive and escalate-only: nothing was relaxed to make room for it, and absence from the list is evidence of nothing at all.
Reproduce it#
The benchmark is a script in the repository, not a slide. It reports packages whose versions are gone separately and never as a detection, for the reason at the top of this post. Run it against a fresh OSV dump and you will get your own numbers; ours are from 2026-08-08.
The corpus, including the cases we miss, is public at depwall/depwall-corpus.