Learning Diverse Representations for Fast Adaptation to Distribution Shift
arXiv:2006.07119
Abstract
The i.i.d. assumption is a useful idealization that underpins many successful approaches to supervised machine learning. However, its violation can lead to models that learn to exploit spurious correlations in the training data, rendering them vulnerable to adversarial interventions, undermining their reliability, and limiting their practical application. To mitigate this problem, we present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task. We propose a notion of diversity based on minimizing the conditional total correlation of final layer representations across models given the label, which we approximate using a variational estimator and minimize using adversarial training. To demonstrate our framework's ability to facilitate rapid adaptation to distribution shift, we train a number of simple classifiers from scratch on the frozen outputs of our models using a small amount of data from the shifted distribution. Under this evaluation protocol, our framework significantly outperforms a baseline trained using the empirical risk minimization principle.
References in corpus (10)
- Disentangling by Factorising
- Determinantal point processes for machine learning
- Illuminating search spaces by mapping elites
- Adversarial Examples Are Not Bugs, They Are Features
- Invariant Risk Minimization
- Out-of-Distribution Generalization via Risk Extrapolation (REx)
- Learning Independent Features with Adversarial Nets for Non-linear ICA
- Diversity Networks: Neural Network Compression Using Determinantal Point Processes
- Invariant Risk Minimization Games
- Recognition in Terra Incognita