Towards falsifiable interpretability research

Abstract

Methods for understanding the decisions of and mechanisms underlying deep neural networks (DNNs) typically rely on building intuition by emphasizing sensory or semantic features of individual examples. For instance, methods aim to visualize the components of an input which are "important" to a network's decision, or to measure the semantic properties of single neurons. Here, we argue that interpretability research suffers from an over-reliance on intuition-based approaches that risk-and in some cases have caused-illusory progress and misleading conclusions. We identify a set of limitations that we argue impede meaningful progress in interpretability research, and examine two popular classes of interpretability methods-saliency and single-neuron-based approaches-that serve as case studies for how overreliance on intuition and lack of falsifiability can undermine interpretability research. To address these concerns, we propose a strategy to address these impediments in the form of a framework for strongly falsifiable interpretability research. We encourage researchers to use their intuitions as a starting point to develop and test clear, falsifiable hypotheses, and hope that our framework yields robust, evidence-based interpretability methods that generate meaningful advances in our understanding of DNNs.

References in corpus (34)

• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• Methods for Interpreting and Understanding Deep Neural Networks
• Learning Important Features Through Propagating Activation Differences
• Dueling Network Architectures for Deep Reinforcement Learning
• Understanding Neural Networks Through Deep Visualization
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• Explainable Machine Learning for Scientific Insights and Discoveries
• Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
• Visualizing and Understanding Recurrent Networks
• SmoothGrad: removing noise by adding noise
• How to Explain Individual Classification Decisions
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• A Benchmark for Interpretability Methods in Deep Neural Networks
• Learning to Generate Reviews and Discovering Sentiment
• Grad-CAM: Why did you say that?
• On the importance of single directions for generalization
• Ablation Studies in Artificial Neural Networks
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
• Visualizing and Understanding Atari Agents
• Understanding Deep Image Representations by Inverting Them
• Revisiting the Importance of Individual Units in CNNs via Ablation
• How Important Is a Neuron?
• TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays
• Understanding trained CNNs by indexing neuron selectivity
• Unsupervised learning of object semantic parts from internal states of CNNs by population encoding
• Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study
• It Takes Two to Tango: Towards Theory of AI's Mind
• The Scientific Method in the Science of Machine Learning
• Towards Interpretable Reinforcement Learning Using Attention Augmented Agents
• Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers
• Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
• Visualizing Uncertainty and Saliency Maps of Deep Convolutional Neural Networks for Medical Imaging Applications
• On the relationship between class selectivity, dimensionality, and robustness

Cited by in corpus (12)

• Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset
• Acquisition of Chess Knowledge in AlphaZero
• Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geoscience
• Explaining Human Activity Recognition with SHAP: Validating Insights with Perturbation and Quantitative Measures
• The effectiveness of feature attribution methods and its correlation with automatic evaluation scores
• How Well do Feature Visualizations Support Causal Understanding of CNN Activations?
• Understanding Consumer Preferences for Explanations Generated by XAI Algorithms
• Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
• Grounding Representation Similarity with Statistical Testing
• Leveraging Sparse Linear Layers for Debuggable Deep Networks
• Two4Two: Evaluating Interpretable Machine Learning - A Synthetic Dataset For Controlled Experiments
• Do Input Gradients Highlight Discriminative Features?