Evaluating the Correctness of Explainable AI Algorithms for Classification

Abstract

Explainable AI has attracted much research attention in recent years with feature attribution algorithms, which compute "feature importance" in predictions, becoming increasingly popular. However, there is little analysis of the validity of these algorithms as there is no "ground truth" in the existing datasets to validate their correctness. In this work, we develop a method to quantitatively evaluate the correctness of XAI algorithms by creating datasets with known explanation ground truth. To this end, we focus on the binary classification problems. String datasets are constructed using formal language derived from a grammar. A string is positive if and only if a certain property is fulfilled. Symbols serving as explanation ground truth in a positive string are part of an explanation if and only if they contributes to fulfilling the property. Two popular feature attribution explainers, Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP), are used in our experiments.We show that: (1) classification accuracy is positively correlated with explanation accuracy; (2) SHAP provides more accurate explanations than LIME; (3) explanation accuracy is negatively correlated with dataset complexity.

References in corpus (9)

• Learning Important Features Through Propagating Activation Differences
• Understanding Black-box Predictions via Influence Functions
• All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously
• Consistent Individualized Feature Attribution for Tree Ensembles
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Explainable AI for Trees: From Local Explanations to Global Understanding
• Metrics for Explainable AI: Challenges and Prospects
• Generating Counterfactual and Contrastive Explanations using SHAP
• Shedding Light on Black Box Machine Learning Algorithms: Development of an Axiomatic Framework to Assess the Quality of Methods that Explain Individual Predictions

Cited by in corpus (2)

• Order in the Court: Explainable AI Methods Prone to Disagreement
• Developing a Fidelity Evaluation Approach for Interpretable Machine Learning