In Search of netUnicorn: A Data-Collection Platform to Develop Generalizable ML Models for Network Security Problems

Abstract

The remarkable success of the use of machine learning-based solutions for network security problems has been impeded by the developed ML models' inability to maintain efficacy when used in different network environments exhibiting different network behaviors. This issue is commonly referred to as the generalizability problem of ML models. The community has recognized the critical role that training datasets play in this context and has developed various techniques to improve dataset curation to overcome this problem. Unfortunately, these methods are generally ill-suited or even counterproductive in the network security domain, where they often result in unrealistic or poor-quality datasets. To address this issue, we propose an augmented ML pipeline that leverages explainable ML tools to guide the network data collection in an iterative fashion. To ensure the data's realism and quality, we require that the new datasets should be endogenously collected in this iterative process, thus advocating for a gradual removal of data-related problems to improve model generalizability. To realize this capability, we develop a data-collection platform, netUnicorn, that takes inspiration from the classic "hourglass" model and is implemented as its "thin waist" to simplify data collection for different learning problems from diverse network environments. The proposed system decouples data-collection intents from the deployment mechanisms and disaggregates these high-level intents into smaller reusable, self-contained tasks. We demonstrate how netUnicorn simplifies collecting data for different learning problems from multiple network environments and how the proposed iterative data collection improves a model's generalizability.

References in corpus (18)

• SMOTE: Synthetic Minority Over-sampling Technique
• Explaining and Harnessing Adversarial Examples
• A Unified Approach to Interpreting Model Predictions
• mixup: Beyond Empirical Risk Minimization
• An Overview of Multi-Task Learning in Deep Neural Networks
• Shortcut Learning in Deep Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Learning under Concept Drift: A Review
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Generalizing Across Domains via Cross-Gradient Training
• InterpretML: A Unified Framework for Machine Learning Interpretability
• Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models
• Why do tree-based models still outperform deep learning on tabular data?
• Evaluation of Machine Learning Classifiers for Zero-Day Intrusion Detection -- An Analysis on CIC-AWS-2018 dataset
• Learning in situ: a randomized experiment in video streaming
• Inferring Streaming Video Quality from Encrypted Traffic: Practical Models and Deployment Experience
• On the Hourglass Model
• Explaining the Predictions of Any Image Classifier via Decision Trees