Qualitatively characterizing neural network optimization problems

Abstract

Training neural networks involves solving large-scale non-convex optimization problems. This task has long been believed to be extremely difficult, with fear of local minima and other obstacles motivating a variety of schemes to improve optimization, such as unsupervised pretraining. However, modern neural networks are able to achieve negligible training error on complex tasks, using only direct training with stochastic gradient descent. We introduce a simple analysis technique to look for evidence that such networks are overcoming local optima. We find that, in fact, on a straight path from initialization to solution, a variety of state of the art neural networks never encounter any significant obstacles.

References in corpus (5)

• Explaining and Harnessing Adversarial Examples
• Recurrent Neural Network Regularization
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Pylearn2: a machine learning research library

Cited by in corpus (113)

• Federated Learning with Non-IID Data
• A continual learning survey: Defying forgetting in classification tasks
• TransMorph: Transformer for unsupervised medical image registration
• A Convergence Theory for Deep Learning via Over-Parameterization
• Visualizing the Loss Landscape of Neural Nets
• Deep Ensembles: A Loss Landscape Perspective
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Averaging Weights Leads to Wider Optima and Better Generalization
• Three Factors Influencing Minima in SGD
• Don't Use Large Mini-Batches, Use Local SGD
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Using transfer learning to detect galaxy mergers
• An Empirical Model of Large-Batch Training
• Optimization for deep learning: theory and algorithms
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• On the Convergence Rate of Training Recurrent Neural Networks
• Gradient Surgery for Multi-Task Learning
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Understanding the Role of Training Regimes in Continual Learning
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• The Global Landscape of Neural Networks: An Overview
• On the Expressive Power of Deep Neural Networks
• On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping
• Measuring the Intrinsic Dimension of Objective Landscapes
• On the loss landscape of a class of deep neural networks with no bad local valleys
• SNIFF: Reverse Engineering of Neural Networks with Fault Attacks
• Auto-Ensemble: An Adaptive Learning Rate Scheduling based Deep Learning Model Ensembling
• Softmax Deep Double Deterministic Policy Gradients
• Understanding the impact of entropy on policy optimization
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• An empirical analysis of the optimization of deep network loss surfaces
• "Convex Until Proven Guilty": Dimension-Free Acceleration of Gradient Descent on Non-Convex Functions
• A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
• The Difficulty of Training Sparse Neural Networks
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• Universal gradient descent
• Visualizing and Understanding the Effectiveness of BERT
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Data efficiency and extrapolation trends in neural network interatomic potentials
• Near-optimal control of dynamical systems with neural ordinary differential equations
• Exploring loss function topology with cyclical learning rates
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Visualizing the PHATE of Neural Networks
• Large Scale Structure of Neural Network Loss Landscapes
• LCA: Loss Change Allocation for Neural Network Training
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• Multi-class Classification without Multi-class Labels
• How degenerate is the parametrization of neural networks with the ReLU activation function?
• Selfish Sparse RNN Training
• Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima
• Stuck in a What? Adventures in Weight Space
• Gradient-only line searches: An Alternative to Probabilistic Line Searches
• Visualizing high-dimensional loss landscapes with Hessian directions
• Multiplicative update rules for accelerating deep learning training and increasing robustness
• The Low-Rank Simplicity Bias in Deep Networks
• Modeling Personalized Item Frequency Information for Next-basket Recommendation
• Full deep neural network training on a pruned weight budget
• Towards Robust Deep Neural Networks
• Understanding Architectures Learnt by Cell-based Neural Architecture Search
• Hierarchical Weight Averaging for Deep Neural Networks
• Taxonomizing local versus global structure in neural network loss landscapes
• Multiscale Hierarchical Convolutional Networks
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• Optimizing Mode Connectivity via Neuron Alignment
• Formal Methods for Autonomous Systems
• Symmetry & critical points for a model shallow neural network
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• GradSign: Model Performance Inference with Theoretical Insights
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Why Learning of Large-Scale Neural Networks Behaves Like Convex Optimization
• Efficient Dictionary Learning with Gradient Descent
• Continual Learning Using Multi-view Task Conditional Neural Networks
• Stochastic gradient descent and fast relaxation to thermodynamic equilibrium: a stochastic control approach
• Evolutionary Augmentation Policy Optimization for Self-supervised Learning
• Spontaneous Symmetry Breaking in Neural Networks
• Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
• Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
• A Comparative Study of Deep Learning Loss Functions for Multi-Label Remote Sensing Image Classification
• Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck
• Intrinsic Geometric Vulnerability of High-Dimensional Artificial Intelligence
• Collaborative Deep Learning Across Multiple Data Centers
• ChemNODE: A Neural Ordinary Differential Equations Approach for Chemical Kinetics Solvers
• Regression as Classification: Influence of Task Formulation on Neural Network Features
• On the topology of solutions to random continuous constraint satisfaction problems
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective
• Towards Deep Learning Models Resistant to Large Perturbations
• Optimization Landscapes of Wide Deep Neural Networks Are Benign
• Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?
• Pre-interpolation loss behaviour in neural networks
• Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide
• Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization
• Fuzzy Logic Interpretation of Quadratic Networks
• Are Saddles Good Enough for Deep Learning?
• Federated Learning as a Mean-Field Game
• In situ TensorView: In situ Visualization of Convolutional Neural Networks
• Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during training
• Expressive Power and Loss Surfaces of Deep Learning Models
• Visualization and Analysis of the Loss Landscape in Graph Neural Networks
• Role of Mixup in Topological Persistence Based Knowledge Distillation for Wearable Sensor Data
• Solving hybrid machine learning tasks by traversing weight space geodesics
• Deep Neural Networks Are Congestion Games: From Loss Landscape to Wardrop Equilibrium and Beyond
• What can linear interpolation of neural network loss landscapes tell us?
• Parabolic Approximation Line Search for DNNs
• Likelihood Landscapes: A Unifying Principle Behind Many Adversarial Defenses
• TATi-Thermodynamic Analytics ToolkIt: TensorFlow-based software for posterior sampling in machine learning applications
• LossPlot: A Better Way to Visualize Loss Landscapes