The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuning
arXiv:2008.10857
Abstract
Biased regularization and fine-tuning are two recent meta-learning approaches. They have been shown to be effective to tackle distributions of tasks, in which the tasks' target vectors are all close to a common meta-parameter vector. However, these methods may perform poorly on heterogeneous environments of tasks, where the complexity of the tasks' distribution cannot be captured by a single meta-parameter vector. We address this limitation by conditional meta-learning, inferring a conditioning function mapping task's side information into a meta-parameter vector that is appropriate for that task at hand. We characterize properties of the environment under which the conditional approach brings a substantial advantage over standard meta-learning and we highlight examples of environments, such as those with multiple clusters, satisfying these properties. We then propose a convex meta-algorithm providing a comparable advantage also in practice. Numerical experiments confirm our theoretical findings.
34 pages; 2 figures
References in corpus (5)
- A New Approach to Collaborative Filtering: Operator Estimation with Spectral Regularization
- Online Meta-Learning
- Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
- Learning-to-Learn Stochastic Gradient Descent with Biased Regularization
- A General Framework for Consistent Structured Prediction with Implicit Loss Embeddings