Learning One-hidden-layer neural networks via Provable Gradient Descent with Random Initialization

Abstract

Although deep learning has shown its powerful performance in many applications, the mathematical principles behind neural networks are still mysterious. In this paper, we consider the problem of learning a one-hidden-layer neural network with quadratic activations. We focus on the under-parameterized regime where the number of hidden units is smaller than the dimension of the inputs. We shall propose to solve the problem via a provable gradient-based method with random initialization. For the non-convex neural networks training problem we reveal that the gradient descent iterates are able to enter a local region that enjoys strong convexity and smoothness within a few iterations, and then provably converges to a globally optimal model at a linear rate with near-optimal sample complexity. We further corroborate our theoretical findings via various experiments.

arXiv commentthe provement need to be corrected

References in corpus (5)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution
• Nonconvex Demixing From Bilinear Measurements
• Blind Over-the-Air Computation and Data Fusion via Provable Wirtinger Flow
• The Global Optimization Geometry of Shallow Linear Neural Networks