The Bouncer Problem: Challenges to Remote Explainability

Abstract

The concept of explainability is envisioned to satisfy society's demands for transparency on machine learning decisions. The concept is simple: like humans, algorithms should explain the rationale behind their decisions so that their fairness can be assessed. While this approach is promising in a local context (e.g. to explain a model during debugging at training time), we argue that this reasoning cannot simply be transposed in a remote context, where a trained model by a service provider is only accessible through its API. This is problematic as it constitutes precisely the target use-case requiring transparency from a societal perspective. Through an analogy with a club bouncer (which may provide untruthful explanations upon customer reject), we show that providing explanations cannot prevent a remote service from lying about the true reasons leading to its decisions. More precisely, we prove the impossibility of remote explainability for single explanations, by constructing an attack on explanations that hides discriminatory features to the querying user. We provide an example implementation of this attack. We then show that the probability that an observer spots the attack, using several explanations for attempting to find incoherences, is low in practical settings. This undermines the very concept of remote explainability in general.

References in corpus (13)

• A Unified Approach to Interpreting Model Predictions
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Model Cards for Model Reporting
• Stealing Machine Learning Models via Prediction APIs
• Fairness Testing: Testing Software for Discrimination
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Fairness in Machine Learning: Lessons from Political Philosophy
• Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation
• Representer Point Selection for Explaining Deep Neural Networks
• Fairwashing: the risk of rationalization
• Fairness Assessment for Artificial Intelligence in Financial Industry
• Logics and practices of transparency and opacity in real-world applications of public sector machine learning

Cited by in corpus (10)

• Adversarial attacks and defenses in explainable artificial intelligence: A survey
• A Critical Survey on Fairness Benefits of Explainable AI
• Machine Learning Partners in Criminal Networks
• When and How to Fool Explainable Models (and Humans) with Adversarial Examples
• Model extraction from counterfactual explanations
• Fooling Partial Dependence via Data Poisoning
• Learning Fair Rule Lists
• It's only fair when I think it's fair: How Gender Bias Alignment Undermines Distributive Fairness in Human-AI Collaboration
• ProtoShotXAI: Using Prototypical Few-Shot Architecture for Explainable AI
• Characterizing the risk of fairwashing