The Global Optimization Geometry of Shallow Linear Neural Networks

Abstract

We examine the squared error loss landscape of shallow linear neural networks. We show---with significantly milder assumptions than previous works---that the corresponding optimization problems have benign geometric properties: there are no spurious local minima and the Hessian at every saddle point has at least one negative eigenvalue. This means that at every saddle point there is a directional negative curvature which algorithms can utilize to further decrease the objective value. These geometric properties imply that many local search algorithms (such as the gradient descent which is widely utilized for training neural networks) can provably solve the training problem with global convergence.

References in corpus (19)

• AMP-Inspired Deep Networks for Sparse Linear Inverse Problems
• Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picture
• Global Optimality of Local Search for Low Rank Matrix Recovery
• Matrix Completion has No Spurious Local Minimum
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• Low-rank Solutions of Linear Matrix Equations via Procrustes Flow
• Global Optimality in Low-rank Matrix Optimization
• Depth Creates No Bad Local Minima
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• When is a Convolutional Filter Easy To Learn?
• Exponentially vanishing sub-optimal local minima in multilayer neural networks
• Accelerated Methods for Non-Convex Optimization
• Global optimality conditions for deep neural networks
• The Global Optimization Geometry of Low-Rank Matrix Optimization
• Convolutional Phase Retrieval via Gradient Descent
• Non-square matrix sensing without spurious local minima via the Burer-Monteiro approach
• Symmetry, Saddle Points, and Global Optimization Landscape of Nonconvex Matrix Factorization
• Deep linear neural networks with arbitrary loss: All local minima are global

Cited by in corpus (3)

• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Decoupling Gating from Linearity
• Learning One-hidden-layer neural networks via Provable Gradient Descent with Random Initialization