Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning

Abstract

Learning-based pattern classifiers, including deep networks, have shown impressive performance in several application domains, ranging from computer vision to cybersecurity. However, it has also been shown that adversarial input perturbations carefully crafted either at training or at test time can easily subvert their predictions. The vulnerability of machine learning to such wild patterns (also referred to as adversarial examples), along with the design of suitable countermeasures, have been investigated in the research field of adversarial machine learning. In this work, we provide a thorough overview of the evolution of this research area over the last ten years and beyond, starting from pioneering, earlier work on the security of non-deep learning algorithms up to more recent work aimed to understand the security properties of deep learning algorithms, in the context of computer vision and cybersecurity tasks. We report interesting connections between these apparently-different lines of work, highlighting common misconceptions related to the security evaluation of machine-learning algorithms. We review the main threat models and attacks defined to this end, and discuss the main limitations of current work, along with the corresponding future challenges towards the design of more secure learning algorithms.

Accepted for publication on Pattern Recognition, 2018

References in corpus (12)

• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Poisoning Attacks against Support Vector Machines
• Stealing Machine Learning Models via Prediction APIs
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Security Evaluation of Pattern Classifiers under Attack
• Is feature selection secure against training data poisoning?
• Adversarial Feature Selection against Evasion Attacks
• Security Analysis of Online Centroid Anomaly Detection
• Certified Defenses for Data Poisoning Attacks
• Randomized Prediction Games for Adversarial Machine Learning
• Statistical Meta-Analysis of Presentation Attacks for Secure Multibiometric Systems

Cited by in corpus (233)

• A Survey on Metaverse: Fundamentals, Security, and Privacy
• Media Forensics and DeepFakes: an overview
• A Unifying Review of Deep and Shallow Anomaly Detection
• Challenges in Deploying Machine Learning: a Survey of Case Studies
• Do ImageNet Classifiers Generalize to ImageNet?
• Robustness May Be at Odds with Accuracy
• Efficient Neural Network Robustness Certification with General Activation Functions
• A Gentle Introduction to Deep Learning for Graphs
• The Role of Machine Learning in Cybersecurity
• Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
• X-ModalNet: A Semi-Supervised Deep Cross-Modal Network for Classification of Remote Sensing Data
• Securing Connected & Autonomous Vehicles: Challenges Posed by Adversarial Machine Learning and The Way Forward
• Adversarial Examples: Opportunities and Challenges
• A Survey of Privacy Attacks in Machine Learning
• A General Framework for Adversarial Examples with Objectives
• Adversarial Example Detection for DNN Models: A Review and Experimental Comparison
• Privacy Risks of Securing Machine Learning Models against Adversarial Examples
• Motivating the Rules of the Game for Adversarial Example Research
• Functionality-preserving Black-box Optimization of Adversarial Windows Malware
• Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning
• Quantum Adversarial Machine Learning
• Trustworthy AI and Robotics and the Implications for the AEC Industry: A Systematic Literature Review and Future Potentials
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• A survey on text generation using generative adversarial networks
• Evaluating Standard Feature Sets Towards Increased Generalisability and Explainability of ML-based Network Intrusion Detection
• I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences
• Adversarial Attack Vulnerability of Medical Image Analysis Systems: Unexplored Factors
• Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training
• Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning
• Do Adversarially Robust ImageNet Models Transfer Better?
• Blind Backdoors in Deep Learning Models
• Adversarial Attacks Against Medical Deep Learning Systems
• On the Effectiveness of System API-Related Information for Android Ransomware Detection
• Recent advances for quantum classifiers
• Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks
• Adversarial Training against Location-Optimized Adversarial Patches
• Towards Adversarial Malware Detection: Lessons Learned from PDF-based Attacks
• A Survey on XAI for 5G and Beyond Security: Technical Aspects, Challenges and Research Directions
• Explaining Vulnerabilities to Adversarial Machine Learning through Visual Analytics
• Phases of methodological research in biostatistics - building the evidence base for new methods
• Humans can decipher adversarial images
• Unlabeled Data Improves Adversarial Robustness
• Adversarial Robustness as a Prior for Learned Representations
• Constrained Concealment Attacks against Reconstruction-based Anomaly Detectors in Industrial Control Systems
• A Tale of Evil Twins: Adversarial Inputs versus Poisoned Models
• Characterizing and evaluating adversarial examples for Offline Handwritten Signature Verification
• POTs: Protective Optimization Technologies
• TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time
• WAF-A-MoLE: Evading Web Application Firewalls through Adversarial Machine Learning
• Robust in Practice: Adversarial Attacks on Quantum Machine Learning
• Understanding and Improving Fast Adversarial Training
• Adversarial Examples - A Complete Characterisation of the Phenomenon
• Invisible Backdoor Attacks on Deep Neural Networks via Steganography and Regularization
• A black-box adversarial attack for poisoning clustering
• Machine Learning Security in Industry: A Quantitative Survey
• Adaptive Metamorphic Testing with Contextual Bandits
• Adversarial classification: An adversarial risk analysis approach
• A Survey of Machine Learning Methods and Challenges for Windows Malware Classification
• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
• Universal Adversarial Audio Perturbations
• Universal Adversarial Examples and Perturbations for Quantum Classifiers
• Multi-SpacePhish: Extending the Evasion-space of Adversarial Attacks against Phishing Website Detectors using Machine Learning
• Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks
• Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
• Quantifying Perceptual Distortion of Adversarial Examples
• Towards a Robust and Trustworthy Machine Learning System Development: An Engineering Perspective
• Voice Mimicry Attacks Assisted by Automatic Speaker Verification
• Ethical Considerations and Statistical Analysis of Industry Involvement in Machine Learning Research
• Mitigating Adversarial Gray-Box Attacks Against Phishing Detectors
• Vulnerabilities of Connectionist AI Applications: Evaluation and Defence
• Towards a Robust Deep Neural Network in Texts: A Survey
• Robustness of Bayesian Neural Networks to Gradient-Based Attacks
• The Threat of Adversarial Attacks on Machine Learning in Network Security -- A Survey
• Explaining Vulnerabilities of Deep Learning to Adversarial Malware Binaries
• Dos and Don'ts of Machine Learning in Computer Security
• Prospective Artificial Intelligence Approaches for Active Cyber Defence
• The RFML Ecosystem: A Look at the Unique Challenges of Applying Deep Learning to Radio Frequency Applications
• The Limitations of Model Uncertainty in Adversarial Settings
• Mitigating Bias in Calibration Error Estimation
• Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks
• Adversarial Machine Learning: Bayesian Perspectives
• Adversarial Detection of Flash Malware: Limitations and Open Issues
• Adversarial Attack on Hierarchical Graph Pooling Neural Networks
• When Causal Intervention Meets Adversarial Examples and Image Masking for Deep Neural Networks
• Reinforcement Learning under Threats
• The Attacker's Perspective on Automatic Speaker Verification: An Overview
• Towards resilient machine learning for ransomware detection
• The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
• Consistency Regularization for Certified Robustness of Smoothed Classifiers
• Turning Federated Learning Systems Into Covert Channels
• On the Adversarial Robustness of Quantized Neural Networks
• Generating Black-Box Adversarial Examples for Text Classifiers Using a Deep Reinforced Model
• Hessian-based toolbox for reliable and interpretable machine learning in physics
• Adversarial Machine Learning in Wireless Communications using RF Data: A Review
• Domain Knowledge Alleviates Adversarial Attacks in Multi-Label Classifiers
• NNoculation: Catching BadNets in the Wild
• Predicting Model Failure using Saliency Maps in Autonomous Driving Systems
• There are No Bit Parts for Sign Bits in Black-Box Attacks
• Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraints
• RED-Attack: Resource Efficient Decision based Attack for Machine Learning
• Evaluating the Robustness of Nearest Neighbor Classifiers: A Primal-Dual Perspective
• Machine Learning for Detecting Data Exfiltration: A Review
• Security and Privacy for Artificial Intelligence: Opportunities and Challenges
• Experimental demonstration of adversarial examples in learning topological phases
• Confidential Machine Learning Computation in Untrusted Environments: A Systems Security Perspective
• Efficient Cyber Attacks Detection in Industrial Control Systems Using Lightweight Neural Networks and PCA
• FooBaR: Fault Fooling Backdoor Attack on Neural Network Training
• Detecting Adversarial Examples through Nonlinear Dimensionality Reduction
• ATHENA: A Framework based on Diverse Weak Defenses for Building Adversarial Defense
• Are Adversarial Perturbations a Showstopper for ML-Based CAD? A Case Study on CNN-Based Lithographic Hotspot Detection
• The Feasibility and Inevitability of Stealth Attacks
• Use the Spear as a Shield: A Novel Adversarial Example based Privacy-Preserving Technique against Membership Inference Attacks
• Adversarial Examples on Object Recognition: A Comprehensive Survey
• Ethical Adversaries: Towards Mitigating Unfairness with Adversarial Machine Learning
• A Marauder's Map of Security and Privacy in Machine Learning
• Certification of embedded systems based on Machine Learning: A survey
• secml: A Python Library for Secure and Explainable Machine Learning
• Fuzzy Unique Image Transformation: Defense Against Adversarial Attacks On Deep COVID-19 Models
• Generating Artificial Outliers in the Absence of Genuine Ones -- a Survey
• CryptoNN: Training Neural Networks over Encrypted Data
• Adversarial Binaries for Authorship Identification
• Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks
• Adversarial Robustness Guarantees for Classification with Gaussian Processes
• Adversarial Machine Learning Phases of Matter
• Bayesian Inference with Certifiable Adversarial Robustness
• Denoised Internal Models: a Brain-Inspired Autoencoder against Adversarial Attacks
• A Visual Analytics Framework for Adversarial Text Generation
• An Optimal Control View of Adversarial Machine Learning
• Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training
• On Data Augmentation and Adversarial Risk: An Empirical Analysis
• Adversarial Security Attacks and Perturbations on Machine Learning and Deep Learning Methods
• Opponent Aware Reinforcement Learning
• Representing Noisy Image Without Denoising
• Computational Limitations in Robust Classification and Win-Win Results
• A Backdoor Attack against 3D Point Cloud Classifiers
• Developing Future Human-Centered Smart Cities: Critical Analysis of Smart City Security, Interpretability, and Ethical Challenges
• The Intriguing Relation Between Counterfactual Explanations and Adversarial Examples
• Detection Defense Against Adversarial Attacks with Saliency Map
• Protecting Classifiers From Attacks
• Attribute Inference Attacks in Online Multiplayer Video Games: a Case Study on Dota2
• Improved Detection of Adversarial Images Using Deep Neural Networks
• Backdoor Attacks on Federated Learning with Lottery Ticket Hypothesis
• Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?
• Why Adversarial Reprogramming Works, When It Fails, and How to Tell the Difference
• Built-in Vulnerabilities to Imperceptible Adversarial Perturbations
• Model-Based Domain Generalization
• Advancing the Research and Development of Assured Artificial Intelligence and Machine Learning Capabilities
• Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples
• PROVEN: Certifying Robustness of Neural Networks with a Probabilistic Approach
• Proof-of-Learning: Definitions and Practice
• Simple statistical methods for unsupervised brain anomaly detection on MRI are competitive to deep learning methods
• Resilience of Autonomous Vehicle Object Category Detection to Universal Adversarial Perturbations
• The Efficacy of SHIELD under Different Threat Models
• Adversarial shape perturbations on 3D point clouds
• Spanning Attack: Reinforce Black-box Attacks with Unlabeled Data
• Diagnosing and Preventing Instabilities in Recurrent Video Processing
• SUPER-Net: Trustworthy Image Segmentation via Uncertainty Propagation in Encoder-Decoder Networks
• Adversarial Training Helps Transfer Learning via Better Representations
• Pseudo-labeling for Scalable 3D Object Detection
• Adversarial Robustness Guarantees for Gaussian Processes
• Improving Adversarial Robustness via Unlabeled Out-of-Domain Data
• Two Sides of the Same Coin: Learning the Backdoor to Remove the Backdoor
• Meta Adversarial Perturbations
• Towards Adversarially Robust Object Detection
• Grundy Distinguishes Treewidth from Pathwidth
• Markpainting: Adversarial Machine Learning meets Inpainting
• Architecture Selection via the Trade-off Between Accuracy and Robustness
• Omni: Automated Ensemble with Unexpected Models against Adversarial Evasion Attack
• A Mixture Model Based Defense for Data Poisoning Attacks Against Naive Bayes Spam Filters
• Understanding the effect of sparsity on neural networks robustness
• Towards a multi-stakeholder value-based assessment framework for algorithmic systems
• Connecting Lyapunov Control Theory to Adversarial Attacks
• Cross-Layer Strategic Ensemble Defense Against Adversarial Examples
• Pocket Diagnosis: Secure Federated Learning against Poisoning Attack in the Cloud
• Practical Cross-modal Manifold Alignment for Grounded Language
• Bias Busters: Robustifying DL-based Lithographic Hotspot Detectors Against Backdooring Attacks
• Adversarial Heart Attack: Neural Networks Fooled to Segment Heart Symbols in Chest X-Ray Images
• Topological Uncertainty: Monitoring trained neural networks through persistence of activation graphs
• Toward Few-step Adversarial Training from a Frequency Perspective
• Decamouflage: A Framework to Detect Image-Scaling Attacks on Convolutional Neural Networks
• Random Polytope Descriptors
• Provably Robust Metric Learning
• Probabilistic Safety for Bayesian Neural Networks
• Lightweight Distributed Gaussian Process Regression for Online Machine Learning
• Estimating the Robustness of Classification Models by the Structure of the Learned Feature-Space
• De-Pois: An Attack-Agnostic Defense against Data Poisoning Attacks
• Empirically Measuring Concentration: Fundamental Limits on Intrinsic Robustness
• HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks
• Risk Management Framework for Machine Learning Security
• Trojaning Language Models for Fun and Profit
• Heating up decision boundaries: isocapacitory saturation, adversarial scenarios and generalization bounds
• Non-Singular Adversarial Robustness of Neural Networks
• Mischief: A Simple Black-Box Attack Against Transformer Architectures
• Provable Robustness of Adversarial Training for Learning Halfspaces with Noise
• Rethinking Empirical Evaluation of Adversarial Robustness Using First-Order Attack Methods
• Challenging the adversarial robustness of DNNs based on error-correcting output codes
• Lower Bounds on Cross-Entropy Loss in the Presence of Test-time Adversaries
• Defending Regression Learners Against Poisoning Attacks
• Does Symbolic Knowledge Prevent Adversarial Fooling?
• Is Ordered Weighted ${\ell }_1$ Regularized Regression Robust to Adversarial Perturbation? A Case Study on OSCAR
• Identifying and Exploiting Structures for Reliable Deep Learning
• Exploring Adversarial Examples for Efficient Active Learning in Machine Learning Classifiers
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
• When Should You Defend Your Classifier -- A Game-theoretical Analysis of Countermeasures against Adversarial Examples
• Law and Adversarial Machine Learning
• On the Security Risks of AutoML
• Towards Robust Reasoning over Knowledge Graphs
• Adversarial Attacks on Knowledge Graph Embeddings via Instance Attribution Methods
• Rethinking Uncertainty in Deep Learning: Whether and How it Improves Robustness
• The Effect of Prior Lipschitz Continuity on the Adversarial Robustness of Bayesian Neural Networks
• Explainable Adversarial Attacks in Deep Neural Networks Using Activation Profiles
• Unique properties of adversarially trained linear classifiers on Gaussian data
• Adversarial Examples and Metrics
• Towards Quality Assurance of Software Product Lines with Adversarial Configurations
• Polarizing Front Ends for Robust CNNs
• The Effectiveness of Johnson-Lindenstrauss Transform for High Dimensional Optimization With Adversarial Outliers, and the Recovery
• Gradient Methods for Solving Stackelberg Games
• A Separation Result Between Data-oblivious and Data-aware Poisoning Attacks
• A Black-box Adversarial Attack Strategy with Adjustable Sparsity and Generalizability for Deep Image Classifiers
• Defending SVMs against Poisoning Attacks: the Hardness and DBSCAN Approach
• A Variational Inequality Approach to Bayesian Regression Games
• Mental Models of Adversarial Machine Learning
• Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects
• Generalized Likelihood Ratio Test for Adversarially Robust Hypothesis Testing
• Measuring Quality of DNA Sequence Data via Degradation
• On the security relevance of weights in deep learning
• Adversarial Perturbation Intensity Achieving Chosen Intra-Technique Transferability Level for Logistic Regression
• Localizing Adversarial Attacks To Produces More Imperceptible Noise
• Model Extraction and Adversarial Attacks on Neural Networks using Switching Power Information
• FAdeML: Understanding the Impact of Pre-Processing Noise Filtering on Adversarial Machine Learning
• Adversarial Machine Learning for Cybersecurity and Computer Vision: Current Developments and Challenges
• Understanding Adversarial Examples Through Deep Neural Network's Response Surface and Uncertainty Regions