Model-Agnostic Explanations using Minimal Forcing Subsets

Abstract

How can we find a subset of training samples that are most responsible for a specific prediction made by a complex black-box machine learning model? More generally, how can we explain the model's decisions to end-users in a transparent way? We propose a new model-agnostic algorithm to identify a minimal set of training samples that are indispensable for a given model's decision at a particular test point, i.e., the model's decision would have changed upon the removal of this subset from the training dataset. Our algorithm identifies such a set of "indispensable" samples iteratively by solving a constrained optimization problem. Further, we speed up the algorithm through efficient approximations and provide theoretical justification for its performance. To demonstrate the applicability and effectiveness of our approach, we apply it to a variety of tasks including data poisoning detection, training set debugging and understanding loan decisions. The results show that our algorithm is an effective and easy-to-comprehend tool that helps to better understand local model behavior, and therefore facilitates the adoption of machine learning in domains where such understanding is a requisite.

arXiv commentInternational Joint Conference on Neural Networks (IJCNN), 2021

References in corpus (16)

• Towards A Rigorous Science of Interpretable Machine Learning
• Learning Important Features Through Propagating Activation Differences
• Understanding Black-box Predictions via Influence Functions
• How to Explain Individual Classification Decisions
• Supersparse Linear Integer Models for Optimized Medical Scoring Systems
• Prototype selection for interpretable classification
• Data Shapley: Equitable Valuation of Data for Machine Learning
• On the Accuracy of Influence Functions for Measuring Group Effects
• Representer Point Selection for Explaining Deep Neural Networks
• Neuron Shapley: Discovering the Responsible Neurons
• Explaining Data-Driven Decisions made by AI Systems: The Counterfactual Approach
• Counterfactual Visual Explanations
• L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data
• Explaining black box decisions by Shapley cohort refinement
• Training Set Debugging Using Trusted Items
• Efficient computation and analysis of distributional Shapley values