Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods

Abstract

As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic errors and underlying biases of black boxes. In this paper, we demonstrate that post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable. Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation. Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous. Using extensive evaluation with multiple real-world datasets (including COMPAS), we demonstrate how extremely biased (racist) classifiers crafted by our framework can easily fool popular explanation techniques such as LIME and SHAP into generating innocuous explanations which do not reflect the underlying biases.

References in corpus (4)

• Towards A Rigorous Science of Interpretable Machine Learning
• Explaining Explanations in AI
• Explanations can be manipulated and geometry is to blame
• Mapping chemical performance on molecular structures using locally interpretable explanations

Cited by in corpus (22)

• DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications
• Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
• Considerations When Learning Additive Explanations for Black-Box Models
• Model extraction from counterfactual explanations
• Proposed Guidelines for the Responsible Use of Explainable Machine Learning
• Beyond TreeSHAP: Efficient Computation of Any-Order Shapley Interactions for Tree Ensembles
• Efficient Learning of Interpretable Classification Rules
• Adversarial Machine Learning and Cybersecurity: Risks, Challenges, and Legal Implications
• Rationalizing Predictions by Adversarial Information Calibration
• Closed-Form Expressions for Global and Local Interpretation of Tsetlin Machines with Applications to Explaining High-Dimensional Data
• Learning Fair Rule Lists
• SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology prediction
• Unifying local and global model explanations by functional decomposition of low dimensional structures
• Learning Hybrid Interpretable Models: Theory, Taxonomy, and Methods
• Challenging common interpretability assumptions in feature attribution explanations
• Can We Trust Your Explanations? Sanity Checks for Interpreters in Android Malware Analysis
• Explainability in AI Based Applications: A Framework for Comparing Different Techniques
• Extending LIME for Business Process Automation
• Auditing and Debugging Deep Learning Models via Decision Boundaries: Individual-level and Group-level Analysis
• Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization
• Working women and caste in India: A study of social disadvantage using feature attribution
• Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation