Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?

Abstract

Can models with particular structure avoid being biased towards spurious correlation in out-of-distribution (OOD) generalization? Peters et al. (2016) provides a positive answer for linear cases. In this paper, we use a functional modular probing method to analyze deep model structures under OOD setting. We demonstrate that even in biased models (which focus on spurious correlation) there still exist unbiased functional subnetworks. Furthermore, we articulate and demonstrate the functional lottery ticket hypothesis: full network contains a subnetwork that can achieve better OOD performance. We then propose Modular Risk Minimization to solve the subnetwork selection problem. Our algorithm learns the subnetwork structure from a given dataset, and can be combined with any other OOD regularization methods. Experiments on various OOD generalization tasks corroborate the effectiveness of our method.

arXiv commentAccepted to ICML2021 as long talk

References in corpus (32)

• Modularity and community structure in networks
• Categorical Reparameterization with Gumbel-Softmax
• Shortcut Learning in Deep Neural Networks
• Wide Residual Networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Domain Generalization via Invariant Feature Representation
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Robustness May Be at Odds with Accuracy
• Invariant Risk Minimization
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Invariant Models for Causal Transfer Learning
• On Causal and Anticausal Learning
• A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Learning De-biased Representations with Biased Representations
• Learning Robust Representations by Projecting Superficial Statistics Out
• The Risks of Invariant Risk Minimization
• What Do Compressed Deep Neural Networks Forget?
• The Pitfalls of Simplicity Bias in Neural Networks
• In Search of Lost Domain Generalization
• Learning explanations that are hard to vary
• Learning from Failure: Training Debiased Classifier from Biased Classifier
• Understanding the Failure Modes of Out-of-Distribution Generalization
• Characterising Bias in Compressed Models
• Informative Dropout for Robust Representation Learning: A Shape-bias Perspective
• Gradient Starvation: A Learning Proclivity in Neural Networks
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Does Invariant Risk Minimization Capture Invariance?
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
• In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness

Cited by in corpus (5)

• Learning Transferable Parameters for Unsupervised Domain Adaptation
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Robust Lottery Tickets for Pre-trained Language Models
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Learning Modular Structures That Generalize Out-of-Distribution