Large-Scale Meta-Learning with Continual Trajectory Shifting
arXiv:2102.07215
Abstract
Meta-learning of shared initialization parameters has shown to be highly effective in solving few-shot learning tasks. However, extending the framework to many-shot scenarios, which may further enhance its practicality, has been relatively overlooked due to the technical difficulties of meta-learning over long chains of inner-gradient steps. In this paper, we first show that allowing the meta-learners to take a larger number of inner gradient steps better captures the structure of heterogeneous and large-scale task distributions, thus results in obtaining better initialization points. Further, in order to increase the frequency of meta-updates even with the excessively long inner-optimization trajectories, we propose to estimate the required shift of the task-specific parameters with respect to the change of the initialization parameters. By doing so, we can arbitrarily increase the frequency of meta-updates and thus greatly improve the meta-level convergence as well as the quality of the learned initializations. We validate our method on a heterogeneous set of large-scale tasks and show that the algorithm largely outperforms the previous first-order meta-learning methods in terms of both generalization performance and convergence, as well as multi-task learning and fine-tuning baselines.
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Prototypical Networks for Few-shot Learning
- Theoretical Models of Learning to Learn
- Fine-Grained Visual Classification of Aircraft
- A Simple Neural Attentive Meta-Learner
- A Neural Representation of Sketch Drawings
- Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
- On First-Order Meta-Learning Algorithms
- Meta-Learning with Implicit Gradients
- The Shattered Gradients Problem: If resnets are the answer, then what is the question?
- Transferring Knowledge across Learning Processes
- Learning What and Where to Transfer