Feature Attributions and Counterfactual Explanations Can Be Manipulated

Abstract

As machine learning models are increasingly used in critical decision-making settings (e.g., healthcare, finance), there has been a growing emphasis on developing methods to explain model predictions. Such \textit{explanations} are used to understand and establish trust in models and are vital components in machine learning pipelines. Though explanations are a critical piece in these systems, there is little understanding about how they are vulnerable to manipulation by adversaries. In this paper, we discuss how two broad classes of explanations are vulnerable to manipulation. We demonstrate how adversaries can design biased models that manipulate model agnostic feature attribution methods (e.g., LIME \& SHAP) and counterfactual explanations that hill-climb during the counterfactual search (e.g., Wachter's Algorithm \& DiCE) into \textit{concealing} the model's biases. These vulnerabilities allow an adversary to deploy a biased model, yet explanations will not reveal this bias, thereby deceiving stakeholders into trusting the model. We evaluate the manipulations on real world data sets, including COMPAS and Communities \& Crime, and find explanations can be manipulated in practice.

arXiv commentarXiv admin note: text overlap with arXiv:2106.02666

References in corpus (8)

• Axiomatic Attribution for Deep Networks
• Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations
• Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
• SmoothGrad: removing noise by adding noise
• Sanity Checks for Saliency Maps
• Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives
• On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
• A survey of algorithmic recourse: definitions, formulations, solutions, and prospects