Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

Abstract

We study the dynamics of optimization and the generalization properties of one-hidden layer neural networks with quadratic activation function in the over-parametrized regime where the layer width $m$ is larger than the input dimension $d$. We consider a teacher-student scenario where the teacher has the same structure as the student with a hidden layer of smaller width $m^{\ast }\le m$. We describe how the empirical loss landscape is affected by the number $n$ of data samples and the width $m^{\ast }$ of the teacher network. In particular we determine how the probability that there be no spurious minima on the empirical loss depends on $n$, $d$, and $m^{\ast }$, thereby establishing conditions under which the neural network can in principle recover the teacher. We also show that under the same conditions gradient descent dynamics on the empirical loss converges and leads to small generalization error, i.e. it enables recovery in practice. Finally we characterize the time-convergence rate of gradient descent in the limit of a large number of samples. These results are confirmed by numerical experiments.

arXiv comment10 pages, 4 figures + appendix

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• String Method for the Study of Rare Events
• Analytical Solution of the Off-Equilibrium Dynamics of a Long Range Spin-Glass Model
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• The loss surface of deep and wide neural networks
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• Global Optimality in Tensor Factorization, Deep Learning, and Beyond
• On the Computational Efficiency of Training Neural Networks
• Provable limitations of deep learning
• Bad Global Minima Exist and SGD Can Reach Them
• Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval
• Stationary Points of Shallow Neural Networks with Quadratic Activation Function

Cited by in corpus (9)

• Phase Retrieval: From Computational Imaging to Machine Learning
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Numerical Calabi-Yau metrics from holomorphic networks
• Noether: The More Things Change, the More Stay the Same
• Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
• On Expressivity and Trainability of Quadratic Networks
• Understanding How Over-Parametrization Leads to Acceleration: A case of learning a single teacher neuron
• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II