Dynamics of Meta-learning Representation in the Teacher-student Scenario

Abstract

Gradient-based meta-learning algorithms have gained popularity for their ability to train models on new tasks using limited data. Empirical observations indicate that such algorithms are able to learn a shared representation across tasks, which is regarded as a key factor in their success. However, the in-depth theoretical understanding of the learning dynamics and the origin of the shared representation remains underdeveloped. In this work, we investigate the meta-learning dynamics of nonlinear two-layer neural networks trained on streaming tasks in the teacher-student scenario. Through the lens of statistical physics analysis, we characterize the macroscopic behavior of the meta-training processes, the formation of the shared representation, and the generalization ability of the model on new tasks. The analysis also points to the importance of the choice of certain hyperparameters of the learning algorithms.

References in corpus (6)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Self-Averaging and On-line Learning
• Exploring the Function Space of Deep-Learning Machines
• Statistical mechanics of continual learning: variational principle and mean-field potential
• Space of Functions Computed by Deep-Layered Machines