Essentially No Barriers in Neural Network Energy Landscape

Abstract

Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.

In Proceedings of 35th International Conference on Machine Learning (ICML 2018)

References in corpus (6)

• On the Number of Linear Regions of Deep Neural Networks
• Visualizing the Loss Landscape of Neural Nets
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• The loss surface of deep and wide neural networks
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Perspective: Energy Landscapes for Machine Learning

Cited by in corpus (89)

• Deep Ensembles: A Loss Landscape Perspective
• Averaging Weights Leads to Wider Optima and Better Generalization
• What is being transferred in transfer learning?
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Enhancing Discrete Choice Models with Representation Learning
• Optimization for deep learning: theory and algorithms
• GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazing
• Rigging the Lottery: Making All Tickets Winners
• Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• What Are Bayesian Neural Network Posteriors Really Like?
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• Adapting Auxiliary Losses Using Gradient Similarity
• The Global Landscape of Neural Networks: An Overview
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
• Searching the solution landscape by generalized high-index saddle dynamics
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Understanding the impact of entropy on policy optimization
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
• Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints
• Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Data efficiency and extrapolation trends in neural network interatomic potentials
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• Large Scale Structure of Neural Network Loss Landscapes
• Selfish Sparse RNN Training
• Avoiding local minima in Variational Quantum Algorithms with Neural Networks
• Learning Neural Network Subspaces
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
• Loss Patterns of Neural Networks
• On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walks
• Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima
• On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case
• Towards Understanding Generalization in Gradient-Based Meta-Learning
• Linear Mode Connectivity in Multitask and Continual Learning
• Directional Pruning of Deep Neural Networks
• Generating Neural Networks with Neural Networks
• Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
• Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective
• Interference and Generalization in Temporal Difference Learning
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Taxonomizing local versus global structure in neural network loss landscapes
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
• Beyond Random Matrix Theory for Deep Networks
• Using Mode Connectivity for Loss Landscape Analysis
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Translating Diffusion, Wavelets, and Regularisation into Residual Networks
• Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
• Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector
• Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks
• Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Function
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
• The global optimum of shallow neural network is attained by ridgelet transform
• Towards Deep Learning Models Resistant to Large Perturbations
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• SuperNet -- An efficient method of neural networks ensembling
• Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscape
• Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension
• Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization
• The role of a layer in deep neural networks: a Gaussian Process perspective
• GridDehazeNet+: An Enhanced Multi-Scale Network with Intra-Task Knowledge Transfer for Single Image Dehazing
• Sparsification as a Remedy for Staleness in Distributed Asynchronous SGD
• Dissecting a Small Artificial Neural Network
• Isotropic SGD: a Practical Approach to Bayesian Posterior Sampling
• The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
• Predicting the outputs of finite deep neural networks trained with noisy gradients
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• GENNI: Visualising the Geometry of Equivalences for Neural Network Identifiability
• Training Deep Neural Networks via Branch-and-Bound
• When Are Solutions Connected in Deep Networks?
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
• Tighter Sparse Approximation Bounds for ReLU Neural Networks
• LossPlot: A Better Way to Visualize Loss Landscapes
• TATi-Thermodynamic Analytics ToolkIt: TensorFlow-based software for posterior sampling in machine learning applications
• A study of local optima for learning feature interactions using neural networks
• What can linear interpolation of neural network loss landscapes tell us?
• Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during training