NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanations

Abstract

Deep neural networks have achieved state-of-the-art performance on various tasks. However, lack of interpretability and transparency makes it easier for malicious attackers to inject trojan backdoor into the neural networks, which will make the model behave abnormally when a backdoor sample with a specific trigger is input. In this paper, we propose NeuronInspect, a framework to detect trojan backdoors in deep neural networks via output explanation techniques. NeuronInspect first identifies the existence of backdoor attack targets by generating the explanation heatmap of the output layer. We observe that generated heatmaps from clean and backdoored models have different characteristics. Therefore we extract features that measure the attributes of explanations from an attacked model namely: sparse, smooth and persistent. We combine these features and use outlier detection to figure out the outliers, which is the set of attack targets. We demonstrate the effectiveness and efficiency of NeuronInspect on MNIST digit recognition dataset and GTSRB traffic sign recognition dataset. We extensively evaluate NeuronInspect on different attack scenarios and prove better robustness and effectiveness over state-of-the-art trojan backdoor detection techniques Neural Cleanse by a great margin.

arXiv comment8 pages, 13 figures

References in corpus (7)

• Striving for Simplicity: The All Convolutional Net
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• DeepID3: Face Recognition with Very Deep Neural Networks
• Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering
• DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
• TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systems
• Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks

Cited by in corpus (12)

• Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning
• Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review
• Blind Backdoors in Deep Learning Models
• You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion
• TrojanZoo: Towards Unified, Holistic, and Practical Evaluation of Neural Backdoors
• An Embarrassingly Simple Approach for Trojan Attack in Deep Neural Networks
• Poisoning Deep Reinforcement Learning Agents with In-Distribution Triggers
• Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models
• EX-RAY: Distinguishing Injected Backdoor from Natural Features in Neural Networks by Examining Differential Feature Symmetry
• PiDAn: A Coherence Optimization Approach for Backdoor Attack Detection and Mitigation in Deep Neural Networks
• Cassandra: Detecting Trojaned Networks from Adversarial Perturbations
• Towards Practical Deployment-Stage Backdoor Attack on Deep Neural Networks