When and How to Fool Explainable Models (and Humans) with Adversarial Examples

Abstract

Reliable deployment of machine learning models such as neural networks continues to be challenging due to several limitations. Some of the main shortcomings are the lack of interpretability and the lack of robustness against adversarial examples or out-of-distribution inputs. In this exploratory review, we explore the possibilities and limits of adversarial attacks for explainable machine learning models. First, we extend the notion of adversarial examples to fit in explainable machine learning scenarios, in which the inputs, the output classifications and the explanations of the model's decisions are assessed by humans. Next, we propose a comprehensive framework to study whether (and how) adversarial examples can be generated for explainable models under human assessment, introducing and illustrating novel attack paradigms. In particular, our framework considers a wide range of relevant yet often ignored factors such as the type of problem, the user expertise or the objective of the explanations, in order to identify the attack strategies that should be adopted in each scenario to successfully deceive the model (and the human). The intention of these contributions is to serve as a basis for a more rigorous and realistic study of adversarial examples in the field of explainable machine learning.

arXiv commentUpdated version. 43 pages, 9 figures, 4 tables

References in corpus (49)

• Explaining and Harnessing Adversarial Examples
• A Unified Approach to Interpreting Model Predictions
• Intriguing properties of neural networks
• Axiomatic Attribution for Deep Networks
• Striving for Simplicity: The All Convolutional Net
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Understanding Neural Networks Through Deep Visualization
• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Understanding Black-box Predictions via Influence Functions
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Delving into Transferable Adversarial Examples and Black-box Attacks
• How to Explain Individual Classification Decisions
• Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systems
• From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Actionable Recourse in Linear Classification
• Generating Natural Adversarial Examples
• Interpretability of machine learning based prediction models in healthcare
• Synthetic and Natural Noise Both Break Neural Machine Translation
• Black-box Adversarial Attacks with Limited Queries and Information
• Concept Whitening for Interpretable Image Recognition
• On Adversarial Examples for Character-Level Neural Machine Translation
• Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models
• Adversarial Attack Vulnerability of Medical Image Analysis Systems: Unexplored Factors
• Fast Feature Fool: A data independent approach to universal adversarial perturbations
• On the Robustness of Interpretability Methods
• Interpreting Adversarially Trained Convolutional Neural Networks
• Adversarial Images for Variational Autoencoders
• From ImageNet to Image Classification: Contextualizing Progress on Benchmarks
• Adversarial Examples for Semantic Image Segmentation
• Imperceptible Adversarial Attacks on Tabular Data
• Fairwashing Explanations with Off-Manifold Detergent
• On the Connection Between Adversarial Robustness and Saliency Map Interpretability
• Proper Network Interpretability Helps Adversarial Robustness in Classification
• The Bouncer Problem: Challenges to Remote Explainability
• Black-box adversarial attacks using Evolution Strategies
• Interpretable Deep Learning under Fire
• CopyCAT: Taking Control of Neural Policies with Constant Attacks
• Adversarial Attacks for Tabular Data: Application to Fraud Detection and Imbalanced Data
• Probabilistically Robust Recourse: Navigating the Trade-offs between Costs and Robustness in Algorithmic Recourse
• Interpretability is a Kind of Safety: An Interpreter-based Ensemble for Adversary Defense
• A simple defense against adversarial attacks on heatmap explanations
• Foiling Explanations in Deep Neural Networks
• Analyzing the Interpretability Robustness of Self-Explaining Models
• Simple Transparent Adversarial Examples
• Counterfactual Explanations Can Be Manipulated
• On the Veracity of Local, Model-agnostic Explanations in Audio Classification: Targeted Investigations with Adversarial Examples
• Robust Classification of Financial Risk

Cited by in corpus (1)

• AEGIS: A Semantic GAN and Evidential Learning Frameworkfor Robust Adversarial Detection in Vision Sensors