Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering

Abstract

While machine learning (ML) models are being increasingly trusted to make decisions in different and varying areas, the safety of systems using such models has become an increasing concern. In particular, ML models are often trained on data from potentially untrustworthy sources, providing adversaries with the opportunity to manipulate them by inserting carefully crafted samples into the training set. Recent work has shown that this type of attack, called a poisoning attack, allows adversaries to insert backdoors or trojans into the model, enabling malicious behavior with simple external backdoor triggers at inference time and only a blackbox perspective of the model itself. Detecting this type of attack is challenging because the unexpected behavior occurs only when a backdoor trigger, which is known only to the adversary, is present. Model users, either direct users of training data or users of pre-trained model from a catalog, may not guarantee the safe operation of their ML-based system. In this paper, we propose a novel approach to backdoor detection and removal for neural networks. Through extensive experimental results, we demonstrate its effectiveness for neural networks classifying text and images. To the best of our knowledge, this is the first methodology capable of detecting poisonous data crafted to insert backdoors and repairing the model that does not require a verified and trusted dataset.

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Convolutional Neural Networks for Sentence Classification
• Understanding deep learning requires rethinking generalization
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• Generative Poisoning Attack Method Against Neural Networks
• Security Analysis of Online Centroid Anomaly Detection
• Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks

Cited by in corpus (83)

• How To Backdoor Federated Learning
• Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning
• Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review
• TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systems
• Rethinking the Trigger of Backdoor Attack
• STRIP: A Defence Against Trojan Attacks on Deep Neural Networks
• NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanations
• You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion
• Security and Privacy Issues in Deep Learning
• On Certifying Robustness against Backdoor Attacks via Randomized Smoothing
• Explanation-Guided Backdoor Poisoning Attacks Against Malware Classifiers
• Backdoor Scanning for Deep Neural Networks through K-Arm Optimization
• Fawkes: Protecting Privacy against Unauthorized Deep Learning Models
• RAB: Provable Robustness Against Backdoor Attacks
• Entangled Watermarks as a Defense against Model Extraction
• A Little Is Enough: Circumventing Defenses For Distributed Learning
• Label-Consistent Backdoor Attacks
• Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks
• Towards Probabilistic Verification of Machine Unlearning
• Adversarial Unlearning of Backdoors via Implicit Hypergradient
• Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
• A Master Key Backdoor for Universal Impersonation Attack against DNN-based Face Verification
• DeepSweep: An Evaluation Framework for Mitigating DNN Backdoor Attacks using Data Augmentation
• TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents
• Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks
• Natural Backdoor Attack on Text Data
• Handcrafted Backdoors in Deep Neural Networks
• Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Cases
• A backdoor attack against LSTM-based text classification systems
• Adversarial Machine Learning in Wireless Communications using RF Data: A Review
• DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations
• Removing Backdoor-Based Watermarks in Neural Networks with Limited Data
• FedV: Privacy-Preserving Federated Learning over Vertically Partitioned Data
• Poison as a Cure: Detecting & Neutralizing Variable-Sized Backdoor Attacks in Deep Neural Networks
• Rethinking the Backdoor Attacks' Triggers: A Frequency Perspective
• Design and Evaluation of a Multi-Domain Trojan Detection Method on Deep Neural Networks
• HaS-Nets: A Heal and Select Mechanism to Defend DNNs Against Backdoor Attacks for Data Collection Scenarios
• T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classification
• Sequential Triggers for Watermarking of Deep Reinforcement Learning Policies
• Sneaky Spikes: Uncovering Stealthy Backdoor Attacks in Spiking Neural Networks with Neuromorphic Data
• RED-Attack: Resource Efficient Decision based Attack for Machine Learning
• Light Can Hack Your Face! Black-box Backdoor Attack on Face Recognition Systems
• Embedding and Extraction of Knowledge in Tree Ensemble Classifiers
• TBT: Targeted Neural Network Attack with Bit Trojan
• An Embarrassingly Simple Approach for Trojan Attack in Deep Neural Networks
• SPECTRE: Defending Against Backdoor Attacks Using Robust Statistics
• TOP: Backdoor Detection in Neural Networks via Transferability of Perturbation
• Poisoning Deep Reinforcement Learning Agents with In-Distribution Triggers
• FooBaR: Fault Fooling Backdoor Attack on Neural Network Training
• Preventing Unauthorized Use of Proprietary Data: Poisoning for Secure Dataset Release
• Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch
• Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training
• BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning
• EX-RAY: Distinguishing Injected Backdoor from Natural Features in Neural Networks by Examining Differential Feature Symmetry
• FaceHack: Triggering backdoored facial recognition systems using facial characteristics
• Regula Sub-rosa: Latent Backdoor Attacks on Deep Neural Networks
• Revealing Perceptible Backdoors, without the Training Set, via the Maximum Achievable Misclassification Fraction Statistic
• Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineering
• Cassandra: Detecting Trojaned Networks from Adversarial Perturbations
• Explainability Matters: Backdoor Attacks on Medical Imaging
• Founding The Domain of AI Forensics
• NTD: Non-Transferability Enabled Backdoor Detection
• HufuNet: Embedding the Left Piece as Watermark and Keeping the Right Piece for Ownership Verification in Deep Neural Networks
• Robust Federated Learning with Attack-Adaptive Aggregation
• A General Framework for Defending Against Backdoor Attacks via Influence Graph
• Curse or Redemption? How Data Heterogeneity Affects the Robustness of Federated Learning
• KNN-enhanced Deep Learning Against Noisy Labels
• Towards Practical Deployment-Stage Backdoor Attack on Deep Neural Networks
• Topological Detection of Trojaned Neural Networks
• TRIGS: Trojan Identification from Gradient-based Signatures
• SGBA: A Stealthy Scapegoat Backdoor Attack against Deep Neural Networks
• A Framework of Randomized Selection Based Certified Defenses Against Data Poisoning Attacks
• Learning and Certification under Instance-targeted Poisoning
• Incompatibility Clustering as a Defense Against Backdoor Poisoning Attacks
• DeepStreamCE: A Streaming Approach to Concept Evolution Detection in Deep Neural Networks
• On Provable Backdoor Defense in Collaborative Learning
• Adversarial examples are useful too!
• PoisHygiene: Detecting and Mitigating Poisoning Attacks in Neural Networks
• TRAPDOOR: Repurposing backdoors to detect dataset bias in machine learning-based genomic analysis
• Widen The Backdoor To Let More Attackers In
• Reverse Engineering Imperceptible Backdoor Attacks on Deep Neural Networks for Detection and Training Set Cleansing
• L-RED: Efficient Post-Training Detection of Imperceptible Backdoor Attacks without Access to the Training Set
• Scalability vs. Utility: Do We Have to Sacrifice One for the Other in Data Importance Quantification?