Multi-step Estimation for Gradient-based Meta-learning
arXiv:2006.04298
Abstract
Gradient-based meta-learning approaches have been successful in few-shot learning, transfer learning, and a wide range of other domains. Despite its efficacy and simplicity, the burden of calculating the Hessian matrix with large memory footprints is the critical challenge in large-scale applications. To tackle this issue, we propose a simple yet straightforward method to reduce the cost by reusing the same gradient in a window of inner steps. We describe the dynamics of the multi-step estimation in the Lagrangian formalism and discuss how to reduce evaluating second-order derivatives estimating the dynamics. To validate our method, we experiment on meta-transfer learning and few-shot learning tasks for multiple settings. The experiment on meta-transfer emphasizes the applicability of training meta-networks, where other approximations are limited. For few-shot learning, we evaluate time and memory complexities compared with popular baselines. We show that our method significantly reduces training time and memory usage, maintaining competitive accuracies, or even outperforming in some cases.
arXiv comment17 pages, 5 figures
References in corpus (12)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Prototypical Networks for Few-shot Learning
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- SGDR: Stochastic Gradient Descent with Warm Restarts
- A Simple Neural Attentive Meta-Learner
- On First-Order Meta-Learning Algorithms
- Few-Shot Learning with Graph Neural Networks
- Meta-Learning with Implicit Gradients
- Knowledge Transfer with Jacobian Matching
- On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms
- Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
- Learning What and Where to Transfer