Generalisation dynamics of online learning in over-parameterised neural networks

Abstract

Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data. Here we study the generalisation dynamics of two-layer neural networks in a teacher-student setup, where one network, the student, is trained using stochastic gradient descent (SGD) on data generated by another network, called the teacher. We show how for this problem, the dynamics of SGD are captured by a set of differential equations. In particular, we demonstrate analytically that the generalisation error of the student increases linearly with the network size, with other relevant parameters held constant. Our results indicate that achieving good generalisation in neural networks depends on the interplay of at least the algorithm, its learning rate, the model architecture, and the data set.

arXiv comment25 pages, 13 figures

References in corpus (4)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Statistical physics of inference: Thresholds and algorithms
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCA

Cited by in corpus (5)

• Data augmentation instead of explicit regularization
• Distributional Generalization: A New Kind of Generalization
• Generalization in multitask deep neural classifiers: a statistical physics approach
• An analytic theory of shallow networks dynamics for hinge loss classification
• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II