A Fine-Grained Analysis on Distribution Shift

Abstract

Robustness to distribution shifts is critical for deploying machine learning models in the real world. Despite this necessity, there has been little work in defining the underlying mechanisms that cause these shifts and evaluating the robustness of algorithms across multiple, different distribution shifts. To this end, we introduce a framework that enables fine-grained analysis of various distribution shifts. We provide a holistic analysis of current state-of-the-art methods by evaluating 19 distinct methods grouped into five categories across both synthetic and real-world datasets. Overall, we train more than 85K models. Our experimental framework can be easily extended to include new methods, shifts, and datasets. We find, unlike previous work~\citep{Gulrajani20}, that progress has been made over a standard ERM baseline; in particular, pretraining and augmentations (learned or heuristic) offer large gains in many cases. However, the best methods are not consistent over different datasets and shifts.

References in corpus (24)

• Auto-Encoding Variational Bayes
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Learning Transferable Visual Models From Natural Language Supervision
• Learning Transferable Features with Deep Adaptation Networks
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Do ImageNet Classifiers Generalize to ImageNet?
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Invariant Risk Minimization
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Understanding disentangling in $\beta$-VAE
• Improve Unsupervised Domain Adaptation with Mixup Training
• Heterogeneous Domain Generalization via Domain Mixup
• Learning From Noisy Singly-labeled Data
• Just Train Twice: Improving Group Robustness without Training Group Information
• Noise or Signal: The Role of Image Backgrounds in Object Recognition
• On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset
• Model Patching: Closing the Subgroup Performance Gap with Data Augmentation
• Support and Invertibility in Domain-Invariant Representations
• When Unseen Domain Generalization is Unnecessary? Rethinking Data Augmentation
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain

Cited by in corpus (2)

• Domain adaptation under structural causal models
• A benchmark with decomposed distribution shifts for 360 monocular depth estimation