Depth creates no more spurious local minima

Abstract

We show that for any convex differentiable loss, a deep linear network has no spurious local minima as long as it is true for the two layer case. This reduction greatly simplifies the study on the existence of spurious local minima in deep linear networks. When applied to the quadratic loss, our result immediately implies the powerful result in [Kawaguchi 2016]. Further, with the work in [Zhou and Liang 2018], we can remove all the assumptions in [Kawaguchi 2016]. This property holds for more general "multi-tower" linear networks too. Our proof builds on [Laurent and von Brecht 2018] and develops a new perturbation argument to show that any spurious local minimum must have full rank, a structural property which can be useful more generally.

References in corpus (8)

• A Convergence Theory for Deep Learning via Over-Parameterization
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Depth Creates No Bad Local Minima
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Small nonlinearities in activation functions create bad local minima in neural networks

Cited by in corpus (5)

• Optimization for deep learning: theory and algorithms
• Pure and Spurious Critical Points: a Geometric Study of Linear Networks
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
• Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations