On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law

Abstract

Out-of-distribution (OOD) testing is increasingly popular for evaluating a machine learning system's ability to generalize beyond the biases of a training set. OOD benchmarks are designed to present a different joint distribution of data and labels between training and test time. VQA-CP has become the standard OOD benchmark for visual question answering, but we discovered three troubling practices in its current use. First, most published methods rely on explicit knowledge of the construction of the OOD splits. They often rely on ``inverting'' the distribution of labels, e.g. answering mostly 'yes' when the common training answer is 'no'. Second, the OOD test set is used for model selection. Third, a model's in-domain performance is assessed after retraining it on in-domain splits (VQA v2) that exhibit a more balanced distribution of labels. These three practices defeat the objective of evaluating generalization, and put into question the value of methods specifically designed for this dataset. We show that embarrassingly-simple methods, including one that generates answers at random, surpass the state of the art on some question types. We provide short- and long-term solutions to avoid these pitfalls and realize the benefits of OOD evaluation.

References in corpus (11)

• Shortcut Learning in Deep Neural Networks
• Deep Reinforcement Learning that Matters
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
• RUBi: Reducing Unimodal Biases in Visual Question Answering
• Self-Critical Reasoning for Robust Visual Question Answering
• Unshuffling Data for Improved Generalization
• Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
• The Scientific Method in the Science of Machine Learning
• Counterfactual Samples Synthesizing for Robust Visual Question Answering
• Visual Grounding Methods for VQA are Working for the Wrong Reasons!

Cited by in corpus (18)

• How I failed machine learning in medical imaging -- shortcomings and recommendations
• Data and its (dis)contents: A survey of dataset development and use in machine learning research
• LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering
• Unshuffling Data for Improved Generalization
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• Robust Collaborative Filtering to Popularity Distribution Shift
• Counterfactual VQA: A Cause-Effect Look at Language Bias
• Language bias in Visual Question Answering: A Survey and Taxonomy
• Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering
• Greedy Gradient Ensemble for Robust Visual Question Answering
• Distribution-aware Fairness Test Generation
• Understanding and Testing Generalization of Deep Networks on Out-of-Distribution Data
• Overcoming Statistical Shortcuts for Open-ended Visual Counting
• Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems
• Deconfounded Video Moment Retrieval with Causal Intervention
• COVR: A test-bed for Visually Grounded Compositional Generalization with real images
• Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering
• X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering