On the energy landscape of deep networks

Abstract

We introduce "AnnealSGD", a regularized stochastic gradient descent algorithm motivated by an analysis of the energy landscape of a particular class of deep networks with sparse random weights. The loss function of such networks can be approximated by the Hamiltonian of a spherical spin glass with Gaussian coupling. While different from currently-popular architectures such as convolutional ones, spin glasses are amenable to analysis, which provides insights on the topology of the loss function and motivates algorithms to minimize it. Specifically, we show that a regularization term akin to a magnetic field can be modulated with a single scalar parameter to transition the loss function from a complex, non-convex landscape with exponentially many local minima, to a phase with a polynomial number of minima, all the way down to a trivial landscape with a unique minimum. AnnealSGD starts training in the relaxed polynomial regime and gradually tightens the regularization parameter to steer the energy towards the original exponential regime. Even for convolutional neural networks, which are quite unlike sparse random networks, we empirically show that AnnealSGD improves the generalization error using competitive baselines on MNIST and CIFAR-10.

References in corpus (7)

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Compressing Neural Networks with the Hashing Trick
• Group Equivariant Convolutional Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Training Recurrent Neural Networks by Diffusion
• A Scale Mixture Perspective of Multiplicative Noise in Neural Networks

Cited by in corpus (6)

• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Dynamical Instantons and Activated Processes in Mean-Field Glass Models
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• Random active path model of deep neural networks with diluted binary synapses
• Appearance of Random Matrix Theory in Deep Learning
• Building a Telescope to Look Into High-Dimensional Image Spaces