Understanding the Loss Surface of Neural Networks for Binary Classification

Abstract

It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performance and generalization performance. Here we focus on the training performance of single-layered neural networks for binary classification, and provide conditions under which the training error is zero at all local minima of a smooth hinge loss function. Our conditions are roughly in the following form: the neurons have to be strictly convex and the surrogate loss function should be a smooth version of hinge loss. We also provide counterexamples to show that when the loss function is replaced with quadratic loss or logistic loss, the result may not hold.

References in corpus (14)

• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Convergence Analysis of Two-layer Neural Networks with ReLU Activation
• Convergent Learning: Do different neural networks learn the same representations?
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Recovery Guarantees for One-hidden-layer Neural Networks
• Global Optimality in Tensor Factorization, Deep Learning, and Beyond
• Learning ReLUs via Gradient Descent
• Identity Matters in Deep Learning
• When is a Convolutional Filter Easy To Learn?
• Exponentially vanishing sub-optimal local minima in multilayer neural networks
• Global optimality conditions for deep neural networks
• Learning Neural Networks with Two Nonlinear Layers in Polynomial Time
• Theoretical properties of the global optimizer of two layer neural network
• Globally Optimal Training of Generalized Polynomial Neural Networks with Nonlinear Spectral Methods

Cited by in corpus (26)

• Optimization for deep learning: theory and algorithms
• The Global Landscape of Neural Networks: An Overview
• On the loss landscape of a class of deep neural networks with no bad local valleys
• Adding One Neuron Can Eliminate All Bad Local Minima
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• Deep Transfer Learning Based Downlink Channel Prediction for FDD Massive MIMO Systems
• Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity
• Visualized Insights into the Optimization Landscape of Fully Convolutional Networks
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Non-attracting Regions of Local Minima in Deep and Wide Neural Networks
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Are deep ResNets provably better than linear predictors?
• Empirical Loss Landscape Analysis of Neural Network Activation Functions
• Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization
• On The Geometric Analysis of A Quartic-quadratic Optimization Problem under A Spherical Constraint
• ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees
• Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extension
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• On the alpha-loss Landscape in the Logistic Model
• When Are Solutions Connected in Deep Networks?
• WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points