RISE: Randomized Input Sampling for Explanation of Black-box Models

Abstract

Deep neural networks are being used increasingly to automate data analysis and decision making, yet their decision-making process is largely unclear and is difficult to explain to the end users. In this paper, we address the problem of Explainable AI for deep neural networks that take images as input and output a class probability. We propose an approach called RISE that generates an importance map indicating how salient each pixel is for the model's prediction. In contrast to white-box approaches that estimate pixel importance using gradients or other internal network state, RISE works on black-box models. It estimates importance empirically by probing the model with randomly masked versions of the input image and obtaining the corresponding outputs. We compare our approach to state-of-the-art importance extraction methods using both an automatic deletion/insertion metric and a pointing metric based on human-annotated object segments. Extensive experiments on several benchmark datasets show that our approach matches or exceeds the performance of other methods, including white-box approaches. Project page: http://cs-people.bu.edu/vpetsiuk/rise/

Cited by in corpus (51)

• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Understanding the Role of Individual Units in a Deep Neural Network
• CovidAID: COVID-19 Detection Using Chest X-Ray
• Fooling Neural Network Interpretations via Adversarial Model Manipulation
• Impossibility Theorems for Feature Attribution
• Explainable Diabetic Retinopathy Detection and Retinal Image Generation
• NBDT: Neural-Backed Decision Trees
• Visualizing Deep Networks by Optimizing with Integrated Gradients
• Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey
• Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional Networks
• BayLIME: Bayesian Local Interpretable Model-Agnostic Explanations
• Smoothed Geometry for Robust Attribution
• Embedding Deep Networks into Visual Explanations
• Explaining generative diffusion models via visual analysis for interpretable decision-making process
• Where is the Model Looking At?--Concentrate and Explain the Network Attention
• Deeply Explain CNN via Hierarchical Decomposition
• Guided Zoom: Questioning Network Evidence for Fine-grained Classification
• Human Attention in Fine-grained Classification
• Explaining Convolutional Neural Networks through Attribution-Based Input Sampling and Block-Wise Feature Aggregation
• Spatio-Temporal Perturbations for Video Attribution
• LIMEcraft: Handcrafted superpixel selection and inspection for Visual eXplanations
• Evaluating Input Perturbation Methods for Interpreting CNNs and Saliency Map Comparison
• Interpretable and Accurate Fine-grained Recognition via Region Grouping
• Why do These Match? Explaining the Behavior of Image Similarity Models
• Attribution Preservation in Network Compression for Reliable Network Interpretation
• One Explanation is Not Enough: Structured Attention Graphs for Image Classification
• Improving Feature Attribution through Input-specific Network Pruning
• Resisting Out-of-Distribution Data Problem in Perturbation of XAI
• Optimising for Interpretability: Convolutional Dynamic Alignment Networks
• FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks
• Investigating Saturation Effects in Integrated Gradients
• Time Distributed Deep Learning Models for Purely Exogenous Forecasting: Application to Water Table Depth Prediction using Weather Image Time Series
• Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces
• Towards Visually Explaining Video Understanding Networks with Perturbation
• Explaining Image Classifiers using Statistical Fault Localization
• Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability
• iGOS++: Integrated Gradient Optimized Saliency by Bilateral Perturbations
• Interactive Mars Image Content-Based Search with Interpretable Machine Learning
• What's wrong with this video? Comparing Explainers for Deepfake Detection
• Visualizing Color-wise Saliency of Black-Box Image Classification Models
• Iterative and Adaptive Sampling with Spatial Attention for Black-Box Model Explanations
• Explanations for Occluded Images
• Minimal Sufficient Views: A DNN model making predictions with more evidence has higher accuracy
• Gradient Frequency Modulation for Visually Explaining Video Understanding Models
• Explain and Improve: LRP-Inference Fine-Tuning for Image Captioning Models
• Efficient Modelling Across Time of Human Actions and Interactions
• Improving Attribution Methods by Learning Submodular Functions
• Play Fair: Frame Attributions in Video Models
• Reinforcement Explanation Learning
• ST-ABN: Visual Explanation Taking into Account Spatio-temporal Information for Video Recognition