Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization

Abstract

In Domain Generalization (DG) settings, models trained independently on a given set of training domains have notoriously chaotic performance on distribution shifted test domains, and stochasticity in optimization (e.g. seed) plays a big role. This makes deep learning models unreliable in real world settings. We first show that this chaotic behavior exists even along the training optimization trajectory of a single model, and propose a simple model averaging protocol that both significantly boosts domain generalization and diminishes the impact of stochasticity by improving the rank correlation between the in-domain validation accuracy and out-domain test accuracy, which is crucial for reliable early stopping. Taking advantage of our observation, we show that instead of ensembling unaveraged models (that is typical in practice), ensembling moving average models (EoA) from independent runs further boosts performance. We theoretically explain the boost in performance of ensembling and model averaging by adapting the well known Bias-Variance trade-off to the domain generalization setting. On the DomainBed benchmark, when using a pre-trained ResNet-50, this ensemble of averages achieves an average of $68.0\%$, beating vanilla ERM (w/o averaging/ensembling) by $\sim 4\%$, and when using a pre-trained RegNetY-16GF, achieves an average of $76.6\%$, beating vanilla ERM by $6\%$. Our code is available at https://github.com/salesforce/ensemble-of-averages.

arXiv commentAccepted at NeurIPS 2022

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Domain Generalization via Invariant Feature Representation
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Deep Ensembles: A Loss Landscape Perspective
• Billion-scale semi-supervised learning for image classification
• Invariant Risk Minimization
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Averaging Weights Leads to Wider Optima and Better Generalization
• Heterogeneous Domain Generalization via Domain Mixup
• Parallelizing Stochastic Gradient Descent for Least Squares Regression: mini-batching, averaging, and model misspecification
• In Search of Lost Domain Generalization
• Improving out-of-distribution generalization via multi-task self-supervised pretraining
• Semi-Supervised Domain Generalization with Stochastic StyleMatch
• Gradient Matching for Domain Generalization

Cited by in corpus (1)

• POEM: Polarization of Embeddings for Domain-Invariant Representations