Depth with Nonlinearity Creates No Bad Local Minima in ResNets

Abstract

In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are guaranteed to further improve via residual representations. As a result, this paper provides an affirmative answer to an open question stated in a paper in the conference on Neural Information Processing Systems 2018. This paper advances the optimization theory of deep learning only for ResNets and not for other network architectures.

References in corpus (21)

• On the Number of Linear Regions of Deep Neural Networks
• The Loss Surfaces of Multilayer Networks
• The Microsoft 2017 Conversational Speech Recognition System
• Convergence Analysis of Two-layer Neural Networks with ReLU Activation
• The loss surface of deep and wide neural networks
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• Recovery Guarantees for One-hidden-layer Neural Networks
• On the number of response regions of deep feed forward networks with piece-wise linear activations
• Learning One-hidden-layer Neural Networks with Landscape Design
• Learning ReLUs via Gradient Descent
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• Bayesian Optimization with Exponential Convergence
• On the Computational Efficiency of Training Neural Networks
• Identity Matters in Deep Learning
• Exponentially vanishing sub-optimal local minima in multilayer neural networks
• Optimization Landscape and Expressivity of Deep CNNs
• Are ResNets Provably Better than Linear Predictors?
• Learning Neural Networks with Two Nonlinear Layers in Polynomial Time
• Provable Methods for Training Neural Networks with Sparse Connectivity
• Global Continuous Optimization with Error Bound and Fast Convergence
• Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review

Cited by in corpus (22)

• Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology
• Review: Deep Learning in Electron Microscopy
• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• Nonlinear Approximation via Compositions
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Effect of Depth and Width on Local Minima in Deep Learning
• Elimination of All Bad Local Minima in Deep Learning
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
• Weakly-Supervised Action Localization and Action Recognition using Global-Local Attention of 3D CNN
• On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
• Benchmark tests of atom segmentation deep learning models with a consistent dataset
• Every Local Minimum Value is the Global Minimum Value of Induced Model in Non-convex Machine Learning
• Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes
• Are deep ResNets provably better than linear predictors?
• A Recipe for Global Convergence Guarantee in Deep Neural Networks
• Avoiding Spurious Local Minima in Deep Quadratic Networks
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Theoretical Exploration of Flexible Transmitter Model
• ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees
• When Are Solutions Connected in Deep Networks?
• Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations