Zipf-Gramming: Scaling Byte N-Grams Up to Production Sized Malware Corpora

Abstract

A classifier using byte n-grams as features is the only approach we have found fast enough to meet requirements in size (sub 2 MB), speed (multiple GB/s), and latency (sub 10 ms) for deployment in numerous malware detection scenarios. However, we've consistently found that 6-8 grams achieve the best accuracy on our production deployments but have been unable to deploy regularly updated models due to the high cost of finding the top-k most frequent n-grams over terabytes of executable programs. Because the Zipfian distribution well models the distribution of n-grams, we exploit its properties to develop a new top-k n-gram extractor that is up to $35\times$ faster than the previous best alternative. Using our new Zipf-Gramming algorithm, we are able to scale up our production training set and obtain up to 30\% improvement in AUC at detecting new malware. We show theoretically and empirically that our approach will select the top-k items with little error and the interplay between theory and engineering required to achieve these results.

Published in CIKM 2025

References in corpus (10)

• Power-law distributions in empirical data
• Power laws, Pareto distributions and Zipf's law
• How Popular is Your Paper? An Empirical Study of the Citation Distribution
• Survey of Machine Learning Techniques for Malware Analysis
• Automatic Yara Rule Generation Using Biclustering
• EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers
• A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labels
• Transformers for End-to-End InfoSec Tasks: A Feasibility Study
• ClarAVy: A Tool for Scalable and Accurate Malware Family Labeling
• High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates