Causally motivated Shortcut Removal Using Auxiliary Labels
arXiv:2105.06422
Abstract
Shortcut learning, in which models make use of easy-to-represent but unstable associations, is a major failure mode for robust machine learning. We study a flexible, causally-motivated approach to training robust predictors by discouraging the use of specific shortcuts, focusing on a common setting where a robust predictor could achieve optimal \emph{iid} generalization in principle, but is overshadowed by a shortcut predictor in practice. Our approach uses auxiliary labels, typically available at training time, to enforce conditional independences implied by the causal graph. We show both theoretically and empirically that causally-motivated regularization schemes (a) lead to more robust estimators that generalize well under distribution shift, and (b) have better finite sample efficiency compared to usual regularization schemes, even when no shortcut is present. Our analysis highlights important theoretical properties of training techniques commonly used in the causal inference, fairness, and disentanglement literatures. Our code is available at https://github.com/mymakar/causally_motivated_shortcut_removal
References in corpus (23)
- A Unified Approach to Interpreting Model Predictions
- Learning Transferable Features with Deep Adaptation Networks
- Deep Domain Confusion: Maximizing for Domain Invariance
- Shortcut Learning in Deep Neural Networks
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- AutoAugment: Learning Augmentation Policies from Data
- Underspecification Presents Challenges for Credibility in Modern Machine Learning
- Generalisation in humans and deep neural networks
- Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
- Invariant Risk Minimization
- Learning Adversarially Fair and Transferable Representations
- Why do deep convolutional networks generalize so poorly to small image transformations?
- Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
- The Risks of Invariant Risk Minimization
- A Fourier Perspective on Model Robustness in Computer Vision
- An Investigation of Why Overparameterization Exacerbates Spurious Correlations
- Anchor regression: heterogeneous data meets causality
- Understanding the Failure Modes of Out-of-Distribution Generalization
- Is There a Trade-Off Between Fairness and Accuracy? A Perspective Using Mismatched Hypothesis Testing
- Inherent Tradeoffs in Learning Fair Representations
- On the Benefits of Invariance in Neural Networks
- Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix
- Estimation of Bounds on Potential Outcomes For Decision Making