What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoors

Abstract

EXplainable AI (XAI) methods have been proposed to interpret how a deep neural network predicts inputs through model saliency explanations that highlight the parts of the inputs deemed important to arrive a decision at a specific target. However, it remains challenging to quantify correctness of their interpretability as current evaluation approaches either require subjective input from humans or incur high computation cost with automated evaluation. In this paper, we propose backdoor trigger patterns--hidden malicious functionalities that cause misclassification--to automate the evaluation of saliency explanations. Our key observation is that triggers provide ground truth for inputs to evaluate whether the regions identified by an XAI method are truly relevant to its output. Since backdoor triggers are the most important features that cause deliberate misclassification, a robust XAI method should reveal their presence at inference time. We introduce three complementary metrics for systematic evaluation of explanations that an XAI method generates and evaluate seven state-of-the-art model-free and model-specific posthoc methods through 36 models trojaned with specifically crafted triggers using color, shape, texture, location, and size. We discovered six methods that use local explanation and feature relevance fail to completely highlight trigger regions, and only a model-free approach can uncover the entire trigger region.

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Deep Residual Learning for Image Recognition
• Striving for Simplicity: The All Convolutional Net
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• Sanity Checks for Saliency Maps
• Visualizing and Understanding Convolutional Networks
• Towards Robust Interpretability with Self-Explaining Neural Networks
• A Benchmark for Interpretability Methods in Deep Neural Networks
• Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems
• Ablation Studies in Artificial Neural Networks
• The (Un)reliability of saliency methods
• Fooling Neural Network Interpretations via Adversarial Model Manipulation
• Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation
• When Explainability Meets Adversarial Learning: Detecting Adversarial Examples using SHAP Signatures
• Interpretable Deep Learning under Fire

Cited by in corpus (4)

• Explainable Artificial Intelligence (XAI): An Engineering Perspective
• BayLIME: Bayesian Local Interpretable Model-Agnostic Explanations
• Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
• TRAPDOOR: Repurposing backdoors to detect dataset bias in machine learning-based genomic analysis