Neuron Shapley: Discovering the Responsible Neurons

Abstract

We develop Neuron Shapley as a new framework to quantify the contribution of individual neurons to the prediction and performance of a deep network. By accounting for interactions across neurons, Neuron Shapley is more effective in identifying important filters compared to common approaches based on activation patterns. Interestingly, removing just 30 filters with the highest Shapley scores effectively destroys the prediction accuracy of Inception-v3 on ImageNet. Visualization of these few critical filters provides insights into how the network functions. Neuron Shapley is a flexible framework and can be applied to identify responsible neurons in many tasks. We illustrate additional applications of identifying filters that are responsible for biased prediction in facial recognition and filters that are vulnerable to adversarial attacks. Removing these filters is a quick way to repair models. Enabling all these applications is a new multi-arm bandit algorithm that we developed to efficiently estimate Neuron Shapley values.

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Ensemble Adversarial Training: Attacks and Defenses
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Empirical Bernstein Bounds and Sample Variance Penalization
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
• Data Shapley: Equitable Valuation of Data for Machine Learning
• How Important Is a Neuron?
• Bounding the Estimation Error of Sampling-based Shapley Value Approximation
• L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data
• Explaining black box decisions by Shapley cohort refinement
• Analysing Neural Network Topologies: a Game Theoretic Approach
• Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits

Cited by in corpus (17)

• Training Data Influence Analysis and Estimation: A Survey
• Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning
• Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning
• Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation
• Towards Counterfactual and Contrastive Explainability and Transparency of DCNN Image Classifiers
• Efficient computation and analysis of distributional Shapley values
• Improving KernelSHAP: Practical Shapley Value Estimation via Linear Regression
• Improving Fairness in Adaptive Social Exergames via Shapley Bandits
• A Quantitative Perspective on Values of Domain Knowledge for Machine Learning
• Attention Flows are Shapley Value Explanations
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution
• Evaluating the Values of Sources in Transfer Learning
• FW-Shapley: Real-time Estimation of Weighted Shapley Values
• NeuroView: Explainable Deep Network Decision Making
• Model-Agnostic Explanations using Minimal Forcing Subsets
• Dynamic Game Theoretic Neural Optimizer