Unshuffling Data for Improved Generalization

Abstract

Generalization beyond the training distribution is a core challenge in machine learning. The common practice of mixing and shuffling examples when training neural networks may not be optimal in this regard. We show that partitioning the data into well-chosen, non-i.i.d. subsets treated as multiple training environments can guide the learning of models with better out-of-distribution generalization. We describe a training procedure to capture the patterns that are stable across environments while discarding spurious ones. The method makes a step beyond correlation-based learning: the choice of the partitioning allows injecting information about the task that cannot be otherwise recovered from the joint distribution of the training data. We demonstrate multiple use cases with the task of visual question answering, which is notorious for dataset biases. We obtain significant improvements on VQA-CP, using environments built from prior knowledge, existing meta data, or unsupervised clustering. We also get improvements on GQA using annotations of "equivalent questions", and on multi-dataset training (VQA v2 / Visual Genome) by treating them as distinct environments.

References in corpus (25)

• VQA: Visual Question Answering
• Invariant Risk Minimization
• Learning Robust Global Representations by Penalizing Local Predictive Power
• Adversarial Examples for Evaluating Reading Comprehension Systems
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• RUBi: Reducing Unimodal Biases in Visual Question Answering
• UNITER: UNiversal Image-TExt Representation Learning
• Invariant Models for Causal Transfer Learning
• Compositional Attention Networks for Machine Reasoning
• Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Women also Snowboard: Overcoming Bias in Captioning Models
• In Search of Lost Domain Generalization
• On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• Invariant Risk Minimization Games
• Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations
• Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
• Language-Conditioned Graph Networks for Relational Reasoning
• An Empirical Study of Invariant Risk Minimization
• Environment Inference for Invariant Learning
• An Empirical Evaluation of Visual Question Answering for Novel Objects
• Learning Rich Image Region Representation for Visual Question Answering
• Misleading Failures of Partial-input Baselines
• Quantifying and Alleviating the Language Prior Problem in Visual Question Answering

Cited by in corpus (18)

• In Search of Lost Domain Generalization
• On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law
• Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• Counterfactual VQA: A Cause-Effect Look at Language Bias
• An Empirical Study of Invariant Risk Minimization
• Empirical or Invariant Risk Minimization? A Sample Complexity Perspective
• Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
• Latent Causal Invariant Model
• Loss re-scaling VQA: Revisiting the LanguagePrior Problem from a Class-imbalance View
• MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
• Supervising the Transfer of Reasoning Patterns in VQA
• Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions
• Causal Attention for Unbiased Visual Recognition
• Robust Classification under Class-Dependent Domain Shift
• On Invariance Penalties for Risk Minimization
• X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering
• Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits