Critical Points of Neural Networks: Analytical Forms and Landscape Properties

Abstract

Due to the success of deep learning to solving a variety of challenging machine learning tasks, there is a rising interest in understanding loss functions for training neural networks from a theoretical aspect. Particularly, the properties of critical points and the landscape around them are of importance to determine the convergence performance of optimization algorithms. In this paper, we provide full (necessary and sufficient) characterization of the analytical forms for the critical points (as well as global minimizers) of the square loss functions for various neural networks. We show that the analytical forms of the critical points characterize the values of the corresponding loss functions as well as the necessary and sufficient conditions to achieve global minimum. Furthermore, we exploit the analytical forms of the critical points to characterize the landscape properties for the loss functions of these neural networks. One particular conclusion is that: The loss function of linear networks has no spurious local minimum, while the loss function of one-hidden-layer nonlinear networks with ReLU activation function does have local minimum that is not global minimum.

References in corpus (6)

• The Loss Surfaces of Multilayer Networks
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Depth Creates No Bad Local Minima
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• Global optimality conditions for deep neural networks
• Demystifying ResNet

Cited by in corpus (19)

• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Optimization for deep learning: theory and algorithms
• The Global Landscape of Neural Networks: An Overview
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Mathematical Models of Overparameterized Neural Networks
• Understanding Deep Learning via Decision Boundary
• Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
• The Local Elasticity of Neural Networks
• On Connected Sublevel Sets in Deep Learning
• Are deep ResNets provably better than linear predictors?
• Numerically Recovering the Critical Points of a Deep Linear Autoencoder
• Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)
• Pure and Spurious Critical Points: a Geometric Study of Linear Networks
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemes
• The Landscape of Multi-Layer Linear Neural Network From the Perspective of Algebraic Geometry
• Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations