The Loss Surfaces of Multilayer Networks

Abstract

We study the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model under the assumptions of: i) variable independence, ii) redundancy in network parametrization, and iii) uniformity. These assumptions enable us to explain the complexity of the fully decoupled neural network through the prism of the results from random matrix theory. We show that for large-size decoupled networks the lowest critical values of the random loss function form a layered structure and they are located in a well-defined band lower-bounded by the global minimum. The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies in real networks. We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This emphasizes a major difference between large- and small-size networks where for the latter poor quality local minima have non-zero probability of being recovered. Finally, we prove that recovering the global minimum becomes harder as the network size increases and that it is in practice irrelevant as global minimum often leads to overfitting.

References in corpus (6)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Predicting Parameters in Deep Learning
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• The statistics of critical points of Gaussian fields on large-dimensional spaces
• Mean-field theory for a spin-glass model of neural networks: TAP free energy and paramagnetic to spin-glass transition
• Random Matrices and complexity of Spin Glasses

Cited by in corpus (308)

• Geometric deep learning: going beyond Euclidean data
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Ensemble deep learning: A review
• Understanding deep learning requires rethinking generalization
• Understanding Deep Convolutional Networks
• Embedding Watermarks into Deep Neural Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Non-convex Optimization for Machine Learning
• A deep material network for multiscale topology learning and accelerated nonlinear modeling of heterogeneous materials
• Recent Advances in Convolutional Neural Networks
• Systematic evaluation of CNN advances on the ImageNet
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Robust Large Margin Deep Neural Networks
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Averaging Weights Leads to Wider Optima and Better Generalization
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Generalization in Deep Learning
• Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
• Deep learning of dynamics and signal-noise decomposition with time-stepping constraints
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges
• Fast Yet Effective Machine Unlearning
• MLPerf Training Benchmark
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• Online Batch Selection for Faster Training of Neural Networks
• Scaling description of generalization with number of parameters in deep learning
• Asynchronous Stochastic Gradient Descent with Delay Compensation
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Sharp Minima Can Generalize For Deep Nets
• GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazing
• Optimization for deep learning: theory and algorithms
• The Modern Mathematics of Deep Learning
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Gradient Descent Converges to Minimizers
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Overfitting Mechanism and Avoidance in Deep Neural Networks
• The Overlap Gap Property: a Geometric Barrier to Optimizing over Random Structures
• When Are Nonconvex Problems Not Scary?
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Machine learning with neural networks
• Deep Neural Network Ensembles for Time Series Classification
• Global Optimality in Tensor Factorization, Deep Learning, and Beyond
• Implicit Full Waveform Inversion with Deep Neural Representation
• A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems
• Mathematics of Deep Learning
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• The Global Landscape of Neural Networks: An Overview
• Depth Creates No Bad Local Minima
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research
• The Power of Normalization: Faster Evasion of Saddle Points
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Deep learning with Elastic Averaging SGD
• Depth with Nonlinearity Creates No Bad Local Minima in ResNets
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• The Landscape of Empirical Risk for Non-convex Losses
• Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach
• On the loss landscape of a class of deep neural networks with no bad local valleys
• Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
• Adversarial Examples in Modern Machine Learning: A Review
• Learning Activation Functions: A new paradigm for understanding Neural Networks
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Uncovering Causality from Multivariate Hawkes Integrated Cumulants
• Stochastic Cubic Regularization for Fast Nonconvex Optimization
• Adding One Neuron Can Eliminate All Bad Local Minima
• Effect of Depth and Width on Local Minima in Deep Learning
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions
• BlackMarks: Blackbox Multibit Watermarking for Deep Neural Networks
• Understanding Batch Normalization
• Elimination of All Bad Local Minima in Deep Learning
• Local minima in training of neural networks
• Machine learning for gravitational-wave detection: surrogate Wiener filtering for the prediction and optimized cancellation of Newtonian noise at Virgo
• Gossip training for deep learning
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
• Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
• Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study
• Memory formation in jammed hard spheres
• Analysis of Stochastic Gradient Descent in Continuous Time
• A Multiple Filter Based Neural Network Approach to the Extrapolation of Adsorption Energies on Metal Surfaces for Catalysis Applications
• On the Decision Boundary of Deep Neural Networks
• A Survey on Neural Network Interpretability
• A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
• Critical Points of Neural Networks: Analytical Forms and Landscape Properties
• Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• Automated Architecture Design for Deep Neural Networks
• On the diffusion approximation of nonconvex stochastic gradient descent
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• Characterization of Gradient Dominance and Regularity Conditions for Neural Networks
• Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks
• Porcupine Neural Networks: (Almost) All Local Optima are Global
• Binary embeddings with structured hashed projections
• Negative eigenvalues of the Hessian in deep neural networks
• HyperGAN: A Generative Model for Diverse, Performant Neural Networks
• Transport Analysis of Infinitely Deep Neural Network
• Recent advances in deep learning theory
• Deep Learning without Poor Local Minima
• DeepTwist: Learning Model Compression via Occasional Weight Distortion
• Deep Learning in Bioinformatics
• Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Loss Landscapes of Regularized Linear Autoencoders
• The Loss Surfaces of Neural Networks with General Activation Functions
• Warfarin dose estimation on multiple datasets with automated hyperparameter optimisation and a novel software framework
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
• Regularizing Activation Distribution for Training Binarized Deep Networks
• Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
• Towards Optimal Power Control via Ensembling Deep Neural Networks
• Stragglers Are Not Disaster: A Hybrid Federated Learning Algorithm with Delayed Gradients
• LCA: Loss Change Allocation for Neural Network Training
• Hessian-based toolbox for reliable and interpretable machine learning in physics
• Large Scale Structure of Neural Network Loss Landscapes
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• How degenerate is the parametrization of neural networks with the ReLU activation function?
• The Landscape of Deep Learning Algorithms
• Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint
• Generalization in Machine Learning via Analytical Learning Theory
• Eigenvalue distribution of nonlinear models of random matrices
• Homotopy Analysis for Tensor PCA
• Generalization Bounds for Convolutional Neural Networks
• Quantitative Propagation of Chaos for SGD in Wide Neural Networks
• From Dependence to Causation
• Hidden Talents of the Variational Autoencoder
• The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification
• The number of saddles of the spherical $p$-spin model
• A Correspondence Between Random Neural Networks and Statistical Field Theory
• CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
• Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training
• Finding Local Minima via Stochastic Nested Variance Reduction
• Counting stationary points of the loss function in the simplest constrained least-square optimization
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Inference in Deep Networks in High Dimensions
• The Multilinear Structure of ReLU Networks
• Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity
• A spin-glass model for the loss surfaces of generative adversarial networks
• Generating Neural Networks with Neural Networks
• CovNet: Covariance Networks for Functional Data on Multidimensional Domains
• On the Stability of Deep Networks
• Exploring generative atomic models in cryo-EM reconstruction
• Towards Robust Deep Neural Networks
• How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?
• Dynamic Inference with Neural Interpreters
• Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
• Deep learning for time series classification
• Invariance of Weight Distributions in Rectified MLPs
• The Local Elasticity of Neural Networks
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Inductive Bias of Gradient Descent based Adversarial Training on Separable Data
• Is SGD a Bayesian sampler? Well, almost
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Stationary Points of Shallow Neural Networks with Quadratic Activation Function
• SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
• Empirical study of extreme overfitting points of neural networks
• Taxonomizing local versus global structure in neural network loss landscapes
• Trusted Artificial Intelligence: Towards Certification of Machine Learning Applications
• On Connected Sublevel Sets in Deep Learning
• Efficiently avoiding saddle points with zero order methods: No gradients required
• Learning-based Traffic State Reconstruction using Probe Vehicles
• Escaping Saddle Points Faster with Stochastic Momentum
• Learning-based State Reconstruction for a Scalar Hyperbolic PDE under noisy Lagrangian Sensing
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Analytic Network Learning
• A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
• Deeper Image Quality Transfer: Training Low-Memory Neural Networks for 3D Images
• Analysis of Invariance and Robustness via Invertibility of ReLU-Networks
• Saving Gradient and Negative Curvature Computations: Finding Local Minima More Efficiently
• Hessian Eigenspectra of More Realistic Nonlinear Models
• Reducing the Training Time of Neural Networks by Partitioning
• Gaussian Mixture Models for Blended Photometric Redshifts
• Non-attracting Regions of Local Minima in Deep and Wide Neural Networks
• Neural Network Memorization Dissection
• Beyond Random Matrix Theory for Deep Networks
• Catch-Up Mix: Catch-Up Class for Struggling Filters in CNN
• The asymptotic spectrum of the Hessian of DNN throughout training
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Positive-Congruent Training: Towards Regression-Free Model Updates
• Bregman Proximal Framework for Deep Linear Neural Networks
• SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Why Learning of Large-Scale Neural Networks Behaves Like Convex Optimization
• Instant Learning: Parallel Deep Neural Networks and Convolutional Bootstrapping
• Deep Neural Networks for Choice Analysis: Extracting Complete Economic Information for Interpretation
• Automated Polysomnography Analysis for Detection of Non-Apneic and Non-Hypopneic Arousals using Feature Engineering and a Bidirectional LSTM Network
• Glass--like transition described by toppling of stability hierarchy
• On generalization bounds for deep networks based on loss surface implicit regularization
• Subsampling in ensemble Kalman inversion
• A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks
• Information-Theoretic Local Minima Characterization and Regularization
• Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
• First-order methods almost always avoid saddle points: the case of vanishing step-sizes
• Improving Confidence Estimates for Unfamiliar Examples
• Low-rank Bilinear Pooling for Fine-Grained Classification
• Edge of chaos as a guiding principle for modern neural network training
• Pure and Spurious Critical Points: a Geometric Study of Linear Networks
• Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector
• Symmetry Breaking in Symmetric Tensor Decomposition
• Universal characteristics of deep neural network loss surfaces from random matrix theory
• Supervised Deep Neural Networks (DNNs) for Pricing/Calibration of Vanilla/Exotic Options Under Various Different Processes
• Convergence Analyses of Online ADAM Algorithm in Convex Setting and Two-Layer ReLU Neural Network
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Universal scaling laws in the gradient descent training of neural networks
• Avoiding Spurious Local Minima in Deep Quadratic Networks
• Distributed Evaluations: Ending Neural Point Metrics
• A Diffusion Approximation Theory of Momentum SGD in Nonconvex Optimization
• The Representation Theory of Neural Networks
• Towards a theory of machine learning
• The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Deep Curvature Suite
• Escaping Saddle Points for Nonsmooth Weakly Convex Functions via Perturbed Proximal Algorithms
• A Topological Framework for Deep Learning
• Third-order Smoothness Helps: Even Faster Stochastic Optimization Algorithms for Finding Local Minima
• Discovery of Latent Factors in High-dimensional Data Using Tensor Methods
• Intrinsic Geometric Vulnerability of High-Dimensional Artificial Intelligence
• Collective evolution of weights in wide neural networks
• A Continuous Optimisation Benchmark Suite from Neural Network Regression
• LV-ROVER: Lexicon Verified Recognizer Output Voting Error Reduction
• Tangent Space Separability in Feedforward Neural Networks
• A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks
• A Note on Connectivity of Sublevel Sets in Deep Learning
• Explaining Natural Language Processing Classifiers with Occlusion and Language Modeling
• The global optimum of shallow neural network is attained by ridgelet transform
• Abstraction Mechanisms Predict Generalization in Deep Neural Networks
• The loss landscape of deep linear neural networks: a second-order analysis
• Subaging in underparametrized Deep Neural Networks
• WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization
• Global Capacity Measures for Deep ReLU Networks via Path Sampling
• Adversarial Robustness Guarantees for Random Deep Neural Networks
• On the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width
• Trap of Feature Diversity in the Learning of MLPs
• On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks
• A relativistic extension of Hopfield neural networks via the mechanical analogy
• Deep Semi-Random Features for Nonlinear Function Approximation
• Criticality & Deep Learning I: Generally Weighted Nets
• Learning Boolean functions with concentrated spectra
• Quantum algorithm for finding the negative curvature direction in non-convex optimization
• Training Deep Neural Networks by optimizing over nonlocal paths in hyperparameter space
• Error Estimation and Correction from within Neural Network Differential Equation Solvers
• Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extension
• Appearance of Random Matrix Theory in Deep Learning
• Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide
• Generalisation in fully-connected neural networks for time series forecasting
• GridDehazeNet+: An Enhanced Multi-Scale Network with Intra-Task Knowledge Transfer for Single Image Dehazing
• Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization
• Measure, Manifold, Learning, and Optimization: A Theory Of Neural Networks
• Saddle-free Hessian-free Optimization
• Pushing the boundaries of parallel Deep Learning -- A practical approach
• Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networks
• Search Spaces for Neural Model Training
• Induction, Popper, and machine learning
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• A Newton-Based Method for Nonconvex Optimization with Fast Evasion of Saddle Points
• Conceptual capacity and effective complexity of neural networks
• NeuSE: A Neural Snapshot Ensemble Method for Collaborative Filtering
• Achieving Small Test Error in Mildly Overparameterized Neural Networks
• On Architectures for Including Visual Information in Neural Language Models for Image Description
• Investigating the interaction between gradient-only line searches and different activation functions
• Distributed stochastic optimization for deep learning (thesis)
• Novel Uncertainty Framework for Deep Learning Ensembles
• Deep Online Learning with Stochastic Constraints
• Optimizing Shallow Networks for Binary Classification
• Acoustic Model Optimization Based On Evolutionary Stochastic Gradient Descent with Anchors for Automatic Speech Recognition
• Some New Results for Poisson Binomial Models
• Are Saddles Good Enough for Deep Learning?
• High Efficient Reconstruction of Single-shot T2 Mapping from OverLapping-Echo Detachment Planar Imaging Based on Deep Residual Network
• BPGrad: Towards Global Optimality in Deep Learning via Branch and Pruning
• Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian
• Good Classifiers are Abundant in the Interpolating Regime
• On the Second-order Convergence Properties of Random Search Methods
• Immunization of Pruning Attack in DNN Watermarking Using Constant Weight Code
• Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum
• Solving hybrid machine learning tasks by traversing weight space geodesics
• How regularization affects the critical points in linear networks
• Towards Scalable Koopman Operator Learning: Convergence Rates and A Distributed Learning Algorithm
• When Can Neural Networks Learn Connected Decision Regions?
• Escaping Saddle Points with Compressed SGD
• Expressive Power and Loss Surfaces of Deep Learning Models
• Expressive power of outer product manifolds on feed-forward neural networks
• A Deterministic Gradient-Based Approach to Avoid Saddle Points
• Training Deep Neural Networks via Branch-and-Bound
• Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations
• TATi-Thermodynamic Analytics ToolkIt: TensorFlow-based software for posterior sampling in machine learning applications
• When Are Solutions Connected in Deep Networks?
• Function approximation with zonal function networks with activation functions analogous to the rectified linear unit functions
• Large-Dimensional Random Matrix Theory and Its Applications in Deep Learning and Wireless Communications
• Not all parameters are born equal: Attention is mostly what you need
• Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
• Weighted Aggregating Stochastic Gradient Descent for Parallel Deep Learning
• On the Differentially Private Nature of Perturbed Gradient Descent
• The Restricted Isometry of ReLU Networks: Generalization through Norm Concentration
• BN-invariant sharpness regularizes the training model to better generalization
• Universality of Gradient Descent Neural Network Training
• A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning
• Learning Graph Neural Networks with Approximate Gradient Descent
• Algebraically-Informed Deep Networks (AIDN): A Deep Learning Approach to Represent Algebraic Structures