Stronger Data Poisoning Attacks Break Data Sanitization Defenses

Abstract

Machine learning models trained on data from the outside world can be corrupted by data poisoning attacks that inject malicious points into the models' training sets. A common defense against these attacks is data sanitization: first filter out anomalous training points before training the model. In this paper, we develop three attacks that can bypass a broad range of common data sanitization defenses, including anomaly detectors based on nearest neighbors, training loss, and singular-value decomposition. By adding just 3% poisoned data, our attacks successfully increase test error on the Enron spam detection dataset from 3% to 24% and on the IMDB sentiment classification dataset from 12% to 29%. In contrast, existing attacks which do not explicitly account for these data sanitization defenses are defeated by them. Our attacks are based on two ideas: (i) we coordinate our attacks to place poisoned points near one another, and (ii) we formulate each attack as a constrained optimization problem, with constraints designed to ensure that the poisoned points evade detection. As this optimization involves solving an expensive bilevel problem, our three attacks correspond to different ways of approximating this problem, based on influence functions; minimax duality; and the Karush-Kuhn-Tucker (KKT) conditions. Our results underscore the need to develop more robust defenses against data poisoning attacks.

This paper was first published on arXiv in 2018 and has since been edited for clarity

Cited by in corpus (42)

• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• On the Accuracy of Influence Functions for Measuring Group Effects
• Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks
• Poisoning Attacks with Generative Adversarial Nets
• On Adversarial Bias and the Robustness of Fair Machine Learning
• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
• Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching
• Adversarial Examples Make Strong Poisons
• Adversarial Neuron Pruning Purifies Backdoored Deep Models
• Data Poisoning against Differentially-Private Learners: Attacks and Defenses
• Defend Data Poisoning Attacks on Voice Authentication
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing
• DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations
• Influence Functions in Deep Learning Are Fragile
• FR-Train: A Mutual Information-Based Approach to Fair and Robust Training
• Security and Privacy for Artificial Intelligence: Opportunities and Challenges
• RelatIF: Identifying Explanatory Training Examples via Relative Influence
• On Second-Order Group Influence Functions for Black-Box Predictions
• Preventing Unauthorized Use of Proprietary Data: Poisoning for Secure Dataset Release
• Defense Against Reward Poisoning Attacks in Reinforcement Learning
• Multi-label Contrastive Predictive Coding
• Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks
• Regularisation Can Mitigate Poisoning Attacks: A Novel Analysis Based on Multiobjective Bilevel Optimisation
• Model-Targeted Poisoning Attacks with Provable Convergence
• Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training
• Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach
• Distributed generation of privacy preserving data with user customization
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
• Picket: Guarding Against Corrupted Data in Tabular Data during Learning and Inference
• Regularization Can Help Mitigate Poisoning Attacks... with the Right Hyperparameters
• Making Paper Reviewing Robust to Bid Manipulation Attacks
• Responsible AI Challenges in End-to-end Machine Learning
• An Investigation of Data Poisoning Defenses for Online Learning
• Generating private data with user customization
• Classification Auto-Encoder based Detector against Diverse Data Poisoning Attacks
• De-Pois: An Attack-Agnostic Defense against Data Poisoning Attacks
• Gradient-based Data Subversion Attack Against Binary Classifiers
• Excess Capacity and Backdoor Poisoning
• The Effectiveness of Johnson-Lindenstrauss Transform for High Dimensional Optimization With Adversarial Outliers, and the Recovery
• Accumulative Poisoning Attacks on Real-time Data
• A Separation Result Between Data-oblivious and Data-aware Poisoning Attacks
• Defending SVMs against Poisoning Attacks: the Hardness and DBSCAN Approach