Interpretable Deep Learning under Fire

Abstract

Providing explanations for deep neural network (DNN) models is crucial for their use in security-sensitive domains. A plethora of interpretation models have been proposed to help users understand the inner workings of DNNs: how does a DNN arrive at a specific decision for a given input? The improved interpretability is believed to offer a sense of security by involving human in the decision-making process. Yet, due to its data-driven nature, the interpretability itself is potentially susceptible to malicious manipulations, about which little is known thus far. Here we bridge this gap by conducting the first systematic study on the security of interpretable deep learning systems (IDLSes). We show that existing \imlses are highly vulnerable to adversarial manipulations. Specifically, we present ADV^2, a new class of attacks that generate adversarial inputs not only misleading target DNNs but also deceiving their coupled interpretation models. Through empirical evaluation against four major types of IDLSes on benchmark datasets and in security-critical applications (e.g., skin cancer diagnosis), we demonstrate that with ADV^2 the adversary is able to arbitrarily designate an input's prediction and interpretation. Further, with both analytical and empirical evidence, we identify the prediction-interpretation gap as one root cause of this vulnerability -- a DNN and its interpretation model are often misaligned, resulting in the possibility of exploiting both models simultaneously. Finally, we explore potential countermeasures against ADV^2, including leveraging its low transferability and incorporating it in an adversarial training framework. Our findings shed light on designing and operating IDLSes in a more secure and informative fashion, leading to several promising research directions.

arXiv commentUSENIX Security Symposium '20

References in corpus (25)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Ensemble Adversarial Training: Attacks and Defenses
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Visualizing and Understanding Recurrent Networks
• Delving into Transferable Adversarial Examples and Black-box Attacks
• SmoothGrad: removing noise by adding noise
• Poisoning Attacks against Support Vector Machines
• Sanity Checks for Saliency Maps
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
• Is feature selection secure against training data poisoning?
• Spatially Transformed Adversarial Examples
• MagNet: a Two-Pronged Defense against Adversarial Examples
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• The (Un)reliability of saliency methods
• Interpretability via Model Extraction
• Skin Lesion Diagnosis using Ensembles, Unscaled Multi-Crop Evaluation and Loss Weighting
• Automatic Rule Extraction from Long Short Term Memory Networks

Cited by in corpus (11)

• Fooling Neural Network Interpretations via Adversarial Model Manipulation
• Interpreting Adversarial Examples by Activation Promotion and Suppression
• When and How to Fool Explainable Models (and Humans) with Adversarial Examples
• What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoors
• Adversarial Infidelity Learning for Model Interpretation
• A simple defense against adversarial attacks on heatmap explanations
• Interpreting Adversarial Examples with Attributes
• Detecting Localized Adversarial Examples: A Generic Approach using Critical Region Analysis
• Aggregating explanation methods for stable and robust explainability
• Can We Trust Your Explanations? Sanity Checks for Interpreters in Android Malware Analysis
• Heat and Blur: An Effective and Fast Defense Against Adversarial Examples