The loss landscape of deep linear neural networks: a second-order analysis

Abstract

We study the optimization landscape of deep linear neural networks with the square loss. It is known that, under weak assumptions, there are no spurious local minima and no local maxima. However, the existence and diversity of non-strict saddle points, which can play a role in first-order algorithms' dynamics, have only been lightly studied. We go a step further with a full analysis of the optimization landscape at order 2. We characterize, among all critical points, which are global minimizers, strict saddle points, and non-strict saddle points. We enumerate all the associated critical values. The characterization is simple, involves conditions on the ranks of partial matrix products, and sheds some light on global convergence or implicit regularization that have been proved or observed when optimizing linear neural networks. In passing, we provide an explicit parameterization of the set of all global minimizers and exhibit large sets of strict and non-strict saddle points.

References in corpus (26)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• The Loss Surfaces of Multilayer Networks
• Benign Overfitting in Linear Regression
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• A mathematical theory of semantic development in deep neural networks
• Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picture
• How to Escape Saddle Points Efficiently
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• The loss surface of deep and wide neural networks
• Optimization for deep learning: theory and algorithms
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Escaping Saddles with Stochastic Gradients
• The Global Landscape of Neural Networks: An Overview
• Depth Creates No Bad Local Minima
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Surprises in High-Dimensional Ridgeless Least Squares Interpolation
• On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• Width Provably Matters in Optimization for Deep Linear Neural Networks
• From Symmetry to Geometry: Tractable Nonconvex Problems
• Optimization Landscape of Tucker Decomposition
• The Implicit Bias of Depth: How Incremental Learning Drives Generalization
• Global Convergence of Gradient Descent for Deep Linear Residual Networks
• Pure and Spurious Critical Points: a Geometric Study of Linear Networks