Assaying Out-Of-Distribution Generalization in Transfer Learning
arXiv:2207.09239
Abstract
Since out-of-distribution generalization is a generally ill-posed problem, various proxy targets (e.g., calibration, adversarial robustness, algorithmic corruptions, invariance across shifts) were studied across different research programs resulting in different recommendations. While sharing the same aspirational goal, these approaches have never been tested under the same experimental conditions on real data. In this paper, we take a unified view of previous work, highlighting message discrepancies that we address empirically, and providing recommendations on how to measure the robustness of a model and how to improve it. To this end, we collect 172 publicly available dataset pairs for training and out-of-distribution evaluation of accuracy, calibration error, adversarial attacks, environment invariance, and synthetic corruptions. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting. Our findings confirm that in- and out-of-distribution accuracies tend to increase jointly, but show that their relation is largely dataset-dependent, and in general more nuanced and more complex than posited by previous, smaller scale studies.
References in corpus (40)
- Adam: A Method for Stochastic Optimization
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Intriguing properties of neural networks
- A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
- Equality of Opportunity in Supervised Learning
- On Calibration of Modern Neural Networks
- MLP-Mixer: An all-MLP Architecture for Vision
- EfficientNetV2: Smaller Models and Faster Training
- Zero-Shot Text-to-Image Generation
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
- Underspecification Presents Challenges for Credibility in Modern Machine Learning
- Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
- Do ImageNet Classifiers Generalize to ImageNet?
- Intriguing Properties of Vision Transformers
- Fairness in Machine Learning
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Learning Robust Global Representations by Penalizing Local Predictive Power
- Out-of-Distribution Generalization via Risk Extrapolation (REx)
- Exploring the Landscape of Spatial Robustness
- Object-Centric Learning with Slot Attention
- Measuring Robustness to Natural Distribution Shifts in Image Classification
- Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
- Measuring Calibration in Deep Learning
- Foolbox: A Python toolbox to benchmark the robustness of machine learning models
- Interventional Few-Shot Learning
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
- Weakly-Supervised Disentanglement Without Compromises
- Are Transformers More Robust Than CNNs?
- Learning Independent Causal Mechanisms
- Towards Causal Representation Learning
- Learning explanations that are hard to vary
- Domino: Discovering Systematic Errors with Cross-Modal Embeddings
- Plex: Towards Reliability using Pretrained Large Model Extensions
- A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness
- The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
- MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts
- Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift
- Dynamic Inference with Neural Interpreters