Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?

Abstract

We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant beta, given by the sum of the reciprocals of the hidden layer widths. When beta is large, the gradients computed by N at initialization vary wildly. Our approach complements the mean field theory analysis of random networks. From this point of view, we rigorously compute finite width corrections to the statistics of gradients at the edge of chaos.

v3. 18p. 1 fig. Accepted at NIPS 2018

Cited by in corpus (8)

• Data driven approximation of parametrized PDEs by Reduced Basis and Neural Networks
• Products of Many Large Random Matrices and Gradients in Deep Neural Networks
• A Fine-Grained Spectral Perspective on Neural Networks
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Manipulating SGD with Data Ordering Attacks
• Interpreting Neural Networks Using Flip Points
• LEMO: Learn to Equalize for MIMO-OFDM Systems with Low-Resolution ADCs
• The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization