Meta Learning in the Continuous Time Limit

Abstract

In this paper, we establish the ordinary differential equation (ODE) that underlies the training dynamics of Model-Agnostic Meta-Learning (MAML). Our continuous-time limit view of the process eliminates the influence of the manually chosen step size of gradient descent and includes the existing gradient descent training algorithm as a special case that results from a specific discretization. We show that the MAML ODE enjoys a linear convergence rate to an approximate stationary point of the MAML loss function for strongly convex task losses, even when the corresponding MAML loss is non-convex. Moreover, through the analysis of the MAML ODE, we propose a new BI-MAML training algorithm that significantly reduces the computational burden associated with existing MAML training methods. To complement our theoretical findings, we perform empirical experiments to showcase the superiority of our proposed methods with respect to the existing work.

25 pages

References in corpus (16)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Theoretical Models of Learning to Learn
• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• Learning to reinforcement learn
• Meta-Learning with Implicit Gradients
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Towards a Neural Statistician
• A Lyapunov Analysis of Momentum Methods in Optimization
• Online Meta-Learning
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Alpha MAML: Adaptive Model-Agnostic Meta-Learning
• Acceleration via Symplectic Discretization of High-Resolution Differential Equations
• ES-MAML: Simple Hessian-Free Meta Learning
• A Dynamical Systems Perspective on Nesterov Acceleration
• Guided Meta-Policy Search