Entropic gradient descent algorithms and wide flat minima

Abstract

The properties of flat minima in the empirical risk landscape of neural networks have been debated for some time. Increasing evidence suggests they possess better generalization capabilities with respect to sharp ones. First, we discuss Gaussian mixture classification models and show analytically that there exist Bayes optimal pointwise estimators which correspond to minimizers belonging to wide flat regions. These estimators can be found by applying maximum flatness algorithms either directly on the classifier (which is norm independent) or on the differentiable loss function used in learning. Next, we extend the analysis to the deep learning scenario by extensive numerical validations. Using two algorithms, Entropy-SGD and Replicated-SGD, that explicitly include in the optimization objective a non-local flatness measure known as local entropy, we consistently improve the generalization error for common architectures (e.g. ResNet, EfficientNet). An easy to compute flatness measure shows a clear correlation with test accuracy.

arXiv commentICLR 2021 camera-ready

References in corpus (11)

• Improved Regularization of Convolutional Neural Networks with Cutout
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Lookahead Optimizer: k steps forward, 1 step back
• Fantastic Generalization Measures and Where to Find Them
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Shaping the learning landscape in neural networks around wide flat minima
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Parle: parallelizing stochastic gradient descent
• Fast AutoAugment

Cited by in corpus (11)

• The Global Landscape of Neural Networks: An Overview
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Learning through atypical "phase transitions" in overparameterized neural networks
• Deep learning via message passing algorithms based on belief propagation
• Visualizing high-dimensional loss landscapes with Hessian directions
• Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
• When you can't count, sample! Computable entropies beyond equilibrium from basin volumes
• Partial local entropy and anisotropy in deep weight spaces
• FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness
• Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry
• Architecture-Aware Minimization (A${}^2$M): How to Find Flat Minima in Neural Architecture Search