Partial local entropy and anisotropy in deep weight spaces

Abstract

We refine a recently-proposed class of local entropic loss functions by restricting the smoothening regularization to only a subset of weights. The new loss functions are referred to as partial local entropies. They can adapt to the weight-space anisotropy, thus outperforming their isotropic counterparts. We support the theoretical analysis with experiments on image classification tasks performed with multi-layer, fully-connected and convolutional neural networks. The present study suggests how to better exploit the anisotropic nature of deep landscapes and provides direct probes of the shape of the minima encountered by stochastic gradient descent algorithms. As a by-product, we observe an asymptotic dynamical regime at late training times where the temperature of all the layers obeys a common cooling behavior.

arXiv commentv3: added analysis of convolutional networks

References in corpus (18)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Improved Regularization of Convolutional Neural Networks with Cutout
• Opening the Black Box of Deep Neural Networks via Information
• Gibbs States and the Set of Solutions of Random Constraint Satisfaction Problems
• Spin-Glass Theory for Pedestrians
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• General $E(2)$-Equivariant Steerable CNNs
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Deep learning with Elastic Averaging SGD
• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Local entropy as a measure for sampling solutions in Constraint Satisfaction Problems
• Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels
• Mean-field inference methods for neural networks
• Entropic gradient descent algorithms and wide flat minima
• Anomalous diffusion and anisotropic nonlinear Fokker-Planck equation
• A Neural Scaling Law from the Dimension of the Data Manifold
• Stochastic gradient descent with random learning rate

Cited by in corpus (1)

• Stochastic gradient descent with random learning rate