Interpreting Black Box Models via Hypothesis Testing

Abstract

In science and medicine, model interpretations may be reported as discoveries of natural phenomena or used to guide patient treatments. In such high-stakes tasks, false discoveries may lead investigators astray. These applications would therefore benefit from control over the finite-sample error rate of interpretations. We reframe black box model interpretability as a multiple hypothesis testing problem. The task is to discover "important" features by testing whether the model prediction is significantly different from what would be expected if the features were replaced with uninformative counterfactuals. We propose two testing methods: one that provably controls the false discovery rate but which is not yet feasible for large-scale applications, and an approximate testing method which can be applied to real-world data sets. In simulation, both tests have high power relative to existing interpretability methods. When applied to state-of-the-art vision and language models, the framework selects features that intuitively explain model predictions. The resulting explanations have the additional advantage that they are themselves easy to interpret.

arXiv commentFODS 2020

References in corpus (18)

• Explaining and Harnessing Adversarial Examples
• A Unified Approach to Interpreting Model Predictions
• YOLOv3: An Incremental Improvement
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Important Features Through Propagating Activation Differences
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Interpretable Explanations of Black Boxes by Meaningful Perturbation
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Sanity Checks for Saliency Maps
• Controlling the false discovery rate via knockoffs
• Exact post-selection inference, with application to the lasso
• Attention is not Explanation
• Kernel-based Conditional Independence Test and Application in Causal Discovery
• Image Inpainting via Generative Multi-column Convolutional Neural Networks
• On Identifiability in Transformers
• Chest X-ray Inpainting with Deep Generative Models
• Mimic and Classify : A meta-algorithm for Conditional Independence Testing
• Interactive Classification for Deep Learning Interpretation

Cited by in corpus (10)

• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Implementing local-explainability in Gradient Boosting Trees: Feature Contribution
• AGRAMPLIFIER: Defending Federated Learning Against Poisoning Attacks Through Local Update Amplification
• What went wrong and when? Instance-wise Feature Importance for Time-series Models
• LIMEcraft: Handcrafted superpixel selection and inspection for Visual eXplanations
• Computationally Efficient Feature Significance and Importance for Machine Learning Models
• DANCE: Enhancing saliency maps using decoys
• Detecting and Understanding Generalization Barriers for Neural Machine Translation
• Learning to Increase the Power of Conditional Randomization Tests
• Explainability Requires Interactivity