Demystifying the Role of Rule-based Detection in AI Systems for Windows Malware Detection

Abstract

Malware detection increasingly relies on AI systems that integrate signature-based detection with machine learning. However, these components are typically developed and combined in isolation, missing opportunities to reduce data complexity and strengthen defenses against adversarial EXEmples, carefully crafted programs designed to evade detection. Hence, in this work we investigate the influence that signature-based detection exerts on model training, when they are included inside the training pipeline. Specifically, we compare models trained on a comprehensive dataset with an AI system whose machine learning component is trained solely on samples not already flagged by signatures. Our results demonstrate improved robustness to both adversarial EXEmples and temporal data drift, although this comes at the cost of a fixed lower bound on false positives, driven by suboptimal rule selection. We conclude by discussing these limitations and outlining how future research could extend AI-based malware detection to include dynamic analysis, thereby further enhancing system resilience.

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• XGBoost: A Scalable Tree Boosting System
• Functionality-preserving Black-box Optimization of Adversarial Windows Malware
• Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection
• Quo Vadis: Hybrid Machine Learning Meta-Model based on Contextual and Behavioral Malware Representations
• ModSec-Learn: Boosting ModSecurity with Machine Learning
• SLIFER: Investigating Performance and Robustness of Malware Detection Pipelines