Training Transformers for Information Security Tasks: A Case Study on Malicious URL Prediction
arXiv:2011.03040
Abstract
Machine Learning (ML) for information security (InfoSec) utilizes distinct data types and formats which require different treatments during optimization/training on raw data. In this paper, we implement a malicious/benign URL predictor based on a transformer architecture that is trained from scratch. We show that in contrast to conventional natural language processing (NLP) transformers, this model requires a different training approach to work well. Specifically, we show that 1) pre-training on a massive corpus of unlabeled URL data for an auto-regressive task does not readily transfer to malicious/benign prediction but 2) that using an auxiliary auto-regressive loss improves performance when training from scratch. We introduce a method for mixed objective optimization, which dynamically balances contributions from both loss terms so that neither one of them dominates. We show that this method yields performance comparable to that of several top-performing benchmark classifiers.
References in corpus (11)
- Scikit-learn: Machine Learning in Python
- Longformer: The Long-Document Transformer
- Snorkel: Rapid Training Data Creation with Weak Supervision
- Malicious URL Detection using Machine Learning: A Survey
- URLNet: Learning a URL Representation with Deep Learning for Malicious URL Detection
- EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models
- eXpose: A Character-Level Convolutional Neural Network with Embeddings For Detecting Malicious URLs, File Paths and Registry Keys
- I-MAD: Interpretable Malware Detector Using Galaxy Transformer
- Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods
- Automatic Malware Description via Attribute Tagging and Similarity Embedding
- Learning from Context: Exploiting and Interpreting File Path Information for Better Malware Detection