Algorithmic Recourse in the Wild: Understanding the Impact of Data and Model Shifts

Abstract

As predictive models are increasingly being deployed to make a variety of consequential decisions, there is a growing emphasis on designing algorithms that can provide recourse to affected individuals. Existing recourse algorithms function under the assumption that the underlying predictive model does not change. However, models are regularly updated in practice for several reasons including data distribution shifts. In this work, we make the first attempt at understanding how model updates resulting from data distribution shifts impact the algorithmic recourses generated by state-of-the-art algorithms. We carry out a rigorous theoretical and empirical analysis to address the above question. Our theoretical results establish a lower bound on the probability of recourse invalidation due to model shifts, and show the existence of a tradeoff between this invalidation probability and typical notions of "cost" minimized by modern recourse generation algorithms. We experiment with multiple synthetic and real world datasets, capturing different kinds of distribution shifts including temporal shifts, geospatial shifts, and shifts due to data correction. These experiments demonstrate that model updation due to all the aforementioned distribution shifts can potentially invalidate recourses generated by state-of-the-art algorithms. Our findings thus not only expose previously unknown flaws in the current recourse generation paradigm, but also pave the way for fundamentally rethinking the design and development of recourse generation algorithms.

References in corpus (16)

• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• Understanding Black-box Predictions via Influence Functions
• Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• SmoothGrad: removing noise by adding noise
• Actionable Recourse in Linear Classification
• The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
• Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
• Learning Model-Agnostic Counterfactual Explanations for Tabular Data
• Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems
• Interpretability via Model Extraction
• Model-Agnostic Counterfactual Explanations for Consequential Decisions
• Algorithmic recourse under imperfect causal knowledge: a probabilistic approach
• Generating Counterfactual and Contrastive Explanations using SHAP
• GeCo: Quality Counterfactual Explanations in Real Time

Cited by in corpus (6)

• A survey of algorithmic recourse: definitions, formulations, solutions, and prospects
• Towards Robust and Reliable Algorithmic Recourse
• Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutions
• Setting the Right Expectations: Algorithmic Recourse Over Time
• A Survey on the Robustness of Feature Importance and Counterfactual Explanations
• DataExposer: Exposing Disconnect between Data and Systems