Nebula: Self-Attention for Dynamic Malware Analysis

Abstract

Dynamic analysis enables detecting Windows malware by executing programs in a controlled environment and logging their actions. Previous work has proposed training machine learning models, i.e., convolutional and long short-term memory networks, on homogeneous input features like runtime APIs to either detect or classify malware, neglecting other relevant information coming from heterogeneous data like network and file operations. To overcome these issues, we introduce Nebula, a versatile, self-attention Transformer-based neural architecture that generalizes across different behavioral representations and formats, combining diverse information from dynamic log reports. Nebula is composed by several components needed to tokenize, filter, normalize and encode data to feed the transformer architecture. We firstly perform a comprehensive ablation study to evaluate their impact on the performance of the whole system, highlighting which components can be used as-is, and which must be enriched with specific domain knowledge. We perform extensive experiments on both malware detection and classification tasks, using three datasets acquired from different dynamic analyses platforms, show that, on average, Nebula outperforms state-of-the-art models at low false positive rates, with a peak of 12% improvement. Moreover, we showcase how self-supervised learning pre-training matches the performance of fully-supervised models with only 20% of training data, and we inspect the output of Nebula through explainable AI techniques, pinpointing how attention is focusing on specific tokens correlated to malicious activities of malware families. To foster reproducibility, we open-source our findings and models at https://github.com/dtrizna/nebula.

arXiv comment18 pages, 7 figures, 12 tables, preprint, in review

References in corpus (7)

• Scaling Laws for Neural Language Models
• Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection
• An Empirical Assessment of Endpoint Security Systems Against Advanced Persistent Threats Attack Vectors
• I-MAD: Interpretable Malware Detector Using Galaxy Transformer
• Quo Vadis: Hybrid Machine Learning Meta-Model based on Contextual and Behavioral Malware Representations
• Transformers for End-to-End InfoSec Tasks: A Feasibility Study
• Avast-CTU Public CAPE Dataset

Cited by in corpus (2)

• SLIFER: Investigating Performance and Robustness of Malware Detection Pipelines
• Bytes to Threats: Byte-Sequence Transformer For Malware Detection