Neural Networks for Learning Counterfactual G-Invariances from Single Environments
arXiv:2104.10105
Abstract
Despite -- or maybe because of -- their astonishing capacity to fit data, neural networks are believed to have difficulties extrapolating beyond training data distribution. This work shows that, for extrapolations based on finite transformation groups, a model's inability to extrapolate is unrelated to its capacity. Rather, the shortcoming is inherited from a learning hypothesis: Examples not explicitly observed with infinitely many training examples have underspecified outcomes in the learner's model. In order to endow neural networks with the ability to extrapolate over group transformations, we introduce a learning framework counterfactually-guided by the learning hypothesis that any group invariance to (known) transformation groups is mandatory even without evidence, unless the learner deems it inconsistent with the training data. Unlike existing invariance-driven methods for (counterfactual) extrapolations, this framework allows extrapolations from a single environment. Finally, we introduce sequence and image extrapolation tasks that validate our framework and showcase the shortcomings of traditional approaches.
arXiv commentICLR 2021
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Shortcut Learning in Deep Neural Networks
- Domain Generalization via Invariant Feature Representation
- Group Equivariant Convolutional Networks
- Underspecification Presents Challenges for Credibility in Modern Machine Learning
- Invariant Risk Minimization
- Causal Effect Inference with Deep Latent-Variable Models
- Out-of-Distribution Generalization via Risk Extrapolation (REx)
- Invariant Models for Causal Transfer Learning
- Causality for Machine Learning
- The Risks of Invariant Risk Minimization
- Learning Independent Causal Mechanisms
- Causal Discovery from Changes
- What Counterfactuals Can Be Tested
- Causal Generative Neural Networks
- Group invariance principles for causal generative models
- On the Benefits of Invariance in Neural Networks