Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization

Abstract

In Domain Generalization (DG) settings, models trained independently on a given set of training domains have notoriously chaotic performance on distribution shifted test domains, and stochasticity in optimization (e.g. seed) plays a big role. This makes deep learning models unreliable in real world settings. We first show that this chaotic behavior exists even along the training optimization trajectory of a single model, and propose a simple model averaging protocol that both significantly boosts domain generalization and diminishes the impact of stochasticity by improving the rank correlation between the in-domain validation accuracy and out-domain test accuracy, which is crucial for reliable early stopping. Taking advantage of our observation, we show that instead of ensembling unaveraged models (that is typical in practice), ensembling moving average models (EoA) from independent runs further boosts performance. We theoretically explain the boost in performance of ensembling and model averaging by adapting the well known Bias-Variance trade-off to the domain generalization setting. On the DomainBed benchmark, when using a pre-trained ResNet-50, this ensemble of averages achieves an average of $68.0\%$, beating vanilla ERM (w/o averaging/ensembling) by $\sim 4\%$, and when using a pre-trained RegNetY-16GF, achieves an average of $76.6\%$, beating vanilla ERM by $6\%$. Our code is available at https://github.com/salesforce/ensemble-of-averages.

Accepted at NeurIPS 2022

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Domain Generalization via Invariant Feature Representation
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Billion-scale semi-supervised learning for image classification
• Heterogeneous Domain Generalization via Domain Mixup
• In Search of Lost Domain Generalization
• Semi-Supervised Domain Generalization with Stochastic StyleMatch
• Gradient Matching for Domain Generalization

Cited by in corpus (1)

• POEM: Polarization of Embeddings for Domain-Invariant Representations