How Effective Are NPM Malicious Package Detectors? A Large-Scale Empirical Study

Abstract

The NPM ecosystem faces escalating threats from malicious packages that exploit its open publication model. While numerous detection tools have been proposed, they are evaluated on disparate datasets with inconsistent settings, making cross-tool comparison unreliable and leaving practitioners without clear guidance. We present the first large-scale empirical study of NPM malicious package detection, evaluating 11 tools with 16 variants on a unified benchmark of 6,420 malicious and 7,288 benign packages annotated with 11 behavior categories and 8 evasion techniques. Unlike prior work, we inspect each tool's source code to explain why tools succeed or fail, not merely how often. Our key findings: (1) the precision and recall a tool achieves are structurally determined by how it resolves the ambiguity between code capability and malicious intent, with IntelGuard reaching the best F1 at 95.98% by grounding its judgment in retrieved evidence and GuardDog the best among conventional tools at 93.32%; (2) behavioral coupling amplifies detection signals when behaviors co-occur, raising SAP_DT from 3.2% to 79.3% for the collect-and-exfiltrate chain; (3) 80.3% of malware uses no evasion because the ecosystem lacks mandatory pre-publication scanning; (4) ML degradation is driven by concept convergence rather than concept drift, since malware became simpler and every decision boundary fitted to a corpus ages with it; (5) combination effectiveness equals complementarity minus false-positive introduction, not paradigm diversity. Strategic combinations reach up to 97.21% accuracy and 97.02% F1. We release our benchmark and evaluation framework.