Characterization of Gradient Dominance and Regularity Conditions for Neural Networks

Abstract

The past decade has witnessed a successful application of deep learning to solving many challenging problems in machine learning and artificial intelligence. However, the loss functions of deep neural networks (especially nonlinear networks) are still far from being well understood from a theoretical aspect. In this paper, we enrich the current understanding of the landscape of the square loss functions for three types of neural networks. Specifically, when the parameter matrices are square, we provide an explicit characterization of the global minimizers for linear networks, linear residual networks, and nonlinear networks with one hidden layer. Then, we establish two quadratic types of landscape properties for the square loss of these neural networks, i.e., the gradient dominance condition within the neighborhood of their full rank global minimizers, and the regularity condition along certain directions and within the neighborhood of their global minimizers. These two landscape properties are desirable for the optimization around the global minimizers of the loss function for these neural networks.

References in corpus (7)

• The Loss Surfaces of Multilayer Networks
• Stochastic Variance Reduction for Nonconvex Optimization
• Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees
• The loss surface of deep and wide neural networks
• Convergence Analysis for Rectangular Matrix Completion Using Burer-Monteiro Factorization and Gradient Descent
• Complex-Valued Autoencoders
• Global optimality conditions for deep neural networks

Cited by in corpus (12)

• Stochastic AUC Maximization with Deep Neural Networks
• SpiderBoost and Momentum: Faster Stochastic Variance Reduction Algorithms
• SGD Converges to Global Minimum in Deep Learning via Star-convex Path
• Stagewise Training Accelerates Convergence of Testing Error Over SGD
• New Analysis of Linear Convergence of Gradient-type Methods via Unifying Error Bound Conditions
• History-Gradient Aided Batch Size Adaptation for Variance Reduced Algorithms
• Proximal Gradient Descent-Ascent: Variable Convergence under KŁ Geometry
• Explicit Explore, Exploit, or Escape ($E^4$): near-optimal safety-constrained reinforcement learning in polynomial time
• On the Quadratic Convergence of the Cubic Regularization Method under a Local Error Bound Condition
• Analytical Convergence Regions of Accelerated Gradient Descent in Nonconvex Optimization under Regularity Condition
• Neural Network Training Techniques Regularize Optimization Trajectory: An Empirical Study
• Asynchronous Parallel Nonconvex Optimization Under the Polyak-Lojasiewicz Condition