Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks

Abstract

This paper revisits the so-called vanishing gradient phenomenon, which commonly occurs in deep randomly initialized neural networks. Leveraging an in-depth analysis of neural chains, we first show that vanishing gradients cannot be circumvented when the network width scales with less than O(depth), even when initialized with the popular Xavier and He initializations. Second, we extend the analysis to second-order derivatives and show that random i.i.d. initialization also gives rise to Hessian matrices with eigenspectra that vanish as networks grow in depth. Whenever this happens, optimizers are initialized in a very flat, saddle point-like plateau, which is particularly hard to escape with stochastic gradient descent (SGD) as its escaping time is inversely related to curvature. We believe that this observation is crucial for fully understanding (a) historical difficulties of training deep nets with vanilla SGD, (b) the success of adaptive gradient methods (which naturally adapt to curvature and thus quickly escape flat plateaus) and (c) the effectiveness of modern architectural components like residual connections and normalization layers.

References in corpus (17)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Deep Residual Learning for Image Recognition
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• New insights and perspectives on the natural gradient method
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• The Emergence of Spectral Universality in Deep Networks
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• Gradient Descent Can Take Exponential Time to Escape Saddle Points
• Products of Many Large Random Matrices and Gradients in Deep Neural Networks
• Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
• Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks
• Characterizing Well-Behaved vs. Pathological Deep Neural Networks