Avoiding Spurious Local Minima in Deep Quadratic Networks

Abstract

Despite their practical success, a theoretical understanding of the loss landscape of neural networks has proven challenging due to the high-dimensional, non-convex, and highly nonlinear structure of such models. In this paper, we characterize the training landscape of the mean squared error loss for neural networks with quadratic activation functions. We prove existence of spurious local minima and saddle points which can be escaped easily with probability one when the number of neurons is greater than or equal to the input dimension and the norm of the training samples is used as a regressor. We prove that deep overparameterized neural networks with quadratic activations benefit from similar nice landscape properties. Our theoretical results are independent of data distribution and fill the existing gap in theory for two-layer quadratic neural networks. Finally, we empirically demonstrate convergence to a global minimum for these problems.

arXiv comment36 pages; added deep network experiments, results for population loss

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• The Loss Surfaces of Multilayer Networks
• Global Optimality of Local Search for Low Rank Matrix Recovery
• No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis
• The loss surface of deep and wide neural networks
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Recovery Guarantees for One-hidden-layer Neural Networks
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• On the Computational Efficiency of Training Neural Networks
• Polynomial Regression As an Alternative to Neural Nets
• Depth with Nonlinearity Creates No Bad Local Minima in ResNets
• Effect of Depth and Width on Local Minima in Deep Learning
• On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition
• Stationary Points of Shallow Neural Networks with Quadratic Activation Function
• Universal Approximation with Quadratic Deep Networks

Cited by in corpus (3)

• On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemes
• Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations
• Understanding How Over-Parametrization Leads to Acceleration: A case of learning a single teacher neuron