Explainable Machine Learning for Public Policy: Use Cases, Gaps, and Research Directions

Abstract

Explainability is highly-desired in Machine Learning (ML) systems supporting high-stakes policy decisions in areas such as health, criminal justice, education, and employment. While the field of explainable ML has expanded in recent years, much of this work has not taken real-world needs into account. A majority of proposed methods are designed with \textit{generic} explainability goals without well-defined use-cases or intended end-users and evaluated on simplified tasks, benchmark problems/datasets, or with proxy users (e.g., AMT). We argue that these simplified evaluation settings do not capture the nuances and complexities of real-world applications. As a result, the applicability and effectiveness of this large body of theoretical and methodological work in real-world applications are unclear. In this work, we take steps toward addressing this gap for the domain of public policy. First, we identify the primary use-cases of explainable ML within public policy problems. For each use case, we define the end-users of explanations and the specific goals the explanations have to fulfill. Finally, we map existing work in explainable ML to these use-cases, identify gaps in established capabilities, and propose research directions to fill those gaps to have a practical societal impact through ML. The contribution is 1) a methodology for explainable ML researchers to identify use cases and develop methods targeted at them and 2) using that methodology for the domain of public policy and giving an example for the researchers on developing explainable ML methods that result in real-world impact.

arXiv commentIn press at Data & Policy

References in corpus (25)

• Towards A Rigorous Science of Interpretable Machine Learning
• Learning Important Features Through Propagating Activation Differences
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Understanding Black-box Predictions via Influence Functions
• Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations
• Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
• How to Explain Individual Classification Decisions
• Consistent Individualized Feature Attribution for Tree Ensembles
• Actionable Recourse in Linear Classification
• Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanations
• FACE: Feasible and Actionable Counterfactual Explanations
• One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• The fallacy of evidence based policy
• Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches
• The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
• Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs
• Interpretable Classification Models for Recidivism Prediction
• Model Agnostic Supervised Local Explanations
• How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations
• Algorithmic Recourse: from Counterfactual Explanations to Interventions
• Learning Optimized Risk Scores
• Towards Robust and Reliable Algorithmic Recourse
• Case Study: Predictive Fairness to Reduce Misdemeanor Recidivism Through Social Service Interventions
• Supersparse Linear Integer Models for Interpretable Classification
• Transductive Optimization of Top k Precision

Cited by in corpus (4)

• Bridging the gap: Towards an Expanded Toolkit for AI-driven Decision-Making in the Public Sector
• Beyond Ads: Sequential Decision-Making Algorithms in Law and Public Policy
• Interpretable Machine Learning: Moving From Mythos to Diagnostics
• Assessing AI Explainability: A Usability Study Using a Novel Framework Involving Clinicians