The Global Landscape of Neural Networks: An Overview

Abstract

One of the major concerns for neural network training is that the non-convexity of the associated loss functions may cause bad landscape. The recent success of neural networks suggests that their loss landscape is not too bad, but what specific results do we know about the landscape? In this article, we review recent findings and results on the global landscape of neural networks. First, we point out that wide neural nets may have sub-optimal local minima under certain assumptions. Second, we discuss a few rigorous results on the geometric properties of wide networks such as "no bad basin", and some modifications that eliminate sub-optimal local minima and/or decreasing paths to infinity. Third, we discuss visualization and empirical explorations of the landscape for practical neural nets. Finally, we briefly discuss some convergence results and their relation to landscape results.

16 pages. 8 figures

References in corpus (25)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• The Loss Surfaces of Multilayer Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Visualizing the Loss Landscape of Neural Nets
• On Exact Computation with an Infinitely Wide Neural Net
• Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview
• On Lazy Training in Differentiable Programming
• How to Escape Saddle Points Efficiently
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Qualitatively characterizing neural network optimization problems
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Essentially No Barriers in Neural Network Energy Landscape
• On the Convergence Rate of Training Recurrent Neural Networks
• Shaping the learning landscape in neural networks around wide flat minima
• Depth Creates No Bad Local Minima
• On the loss landscape of a class of deep neural networks with no bad local valleys
• Understanding the Loss Surface of Neural Networks for Binary Classification
• Elimination of All Bad Local Minima in Deep Learning
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• Critical Points of Neural Networks: Analytical Forms and Landscape Properties
• Entropic gradient descent algorithms and wide flat minima
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity

Cited by in corpus (13)

• Deep matrix factorizations
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Mathematical Models of Overparameterized Neural Networks
• Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data
• Taxonomizing local versus global structure in neural network loss landscapes
• Training a Two Layer ReLU Network Analytically
• AdapMTL: Adaptive Pruning Framework for Multitask Learning Model
• The loss landscape of deep linear neural networks: a second-order analysis
• Non-Convex Exact Community Recovery in Stochastic Block Model
• Neural Networks with Complex-Valued Weights Have No Spurious Local Minima
• On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemes
• WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points
• Embedding Principle: a hierarchical structure of loss landscape of deep neural networks