Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines

Abstract

Over-parametrized deep neural networks trained by stochastic gradient descent are successful in performing many tasks of practical relevance. One aspect of over-parametrization is the possibility that the student network has a larger expressivity than the data generating process. In the context of a student-teacher scenario, this corresponds to the so-called over-realizable case, where the student network has a larger number of hidden units than the teacher. For on-line learning of a two-layer soft committee machine in the over-realizable case, we find that the approach to perfect learning occurs in a power-law fashion rather than exponentially as in the realizable case. All student nodes learn and replicate one of the teacher nodes if teacher and student outputs are suitably rescaled.

arXiv comment5 pages, 5 figures

References in corpus (13)

• Machine learning and the physical sciences
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation
• Self-Averaging and On-line Learning
• Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
• Phase transitions in soft-committee machines
• Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysis