Meta-learning for mixed linear regression

Abstract

In modern supervised learning, there are a large number of tasks, but many of them are associated with only a small amount of labeled data. These include data from medical image processing and robotic interaction. Even though each individual task cannot be meaningfully trained in isolation, one seeks to meta-learn across the tasks from past experiences by exploiting some similarities. We study a fundamental question of interest: When can abundant tasks with small data compensate for lack of tasks with big data? We focus on a canonical scenario where each task is drawn from a mixture of $k$ linear regressions, and identify sufficient conditions for such a graceful exchange to hold; The total number of examples necessary with only small data tasks scales similarly as when big data tasks are available. To this end, we introduce a novel spectral approach and show that we can efficiently utilize small data tasks with the help of $\tilde{\Omega }(k^{3/2})$ medium data tasks each with $\tilde{\Omega }(k^{1/2})$ examples.

References in corpus (6)

• Theoretical Models of Learning to Learn
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Solving a Mixture of Many Random Linear Equations by Tensor Decomposition and Alternating Minimization
• Maximum Likelihood Estimation for Learning Populations of Parameters
• Sublinear Optimal Policy Value Estimation in Contextual Bandits