Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
arXiv:2106.02890
Abstract
Can models with particular structure avoid being biased towards spurious correlation in out-of-distribution (OOD) generalization? Peters et al. (2016) provides a positive answer for linear cases. In this paper, we use a functional modular probing method to analyze deep model structures under OOD setting. We demonstrate that even in biased models (which focus on spurious correlation) there still exist unbiased functional subnetworks. Furthermore, we articulate and demonstrate the functional lottery ticket hypothesis: full network contains a subnetwork that can achieve better OOD performance. We then propose Modular Risk Minimization to solve the subnetwork selection problem. Our algorithm learns the subnetwork structure from a given dataset, and can be combined with any other OOD regularization methods. Experiments on various OOD generalization tasks corroborate the effectiveness of our method.
Accepted to ICML2021 as long talk
References in corpus (10)
- Modularity and community structure in networks
- Domain Generalization via Invariant Feature Representation
- On Causal and Anticausal Learning
- A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
- Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
- Learning Robust Representations by Projecting Superficial Statistics Out
- In Search of Lost Domain Generalization
- Informative Dropout for Robust Representation Learning: A Shape-bias Perspective
- Does Invariant Risk Minimization Capture Invariance?
- In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness