Characterizing Well-Behaved vs. Pathological Deep Neural Networks
arXiv:1811.03087
Abstract
We introduce a novel approach, requiring only mild assumptions, for the characterization of deep neural networks at initialization. Our approach applies both to fully-connected and convolutional networks and easily incorporates batch normalization and skip-connections. Our key insight is to consider the evolution with depth of statistical moments of signal and noise, thereby characterizing the presence or absence of pathologies in the hypothesis space encoded by the choice of hyperparameters. We establish: (i) for feedforward networks, with and without batch normalization, the multiplicativity of layer composition inevitably leads to ill-behaved moments and pathologies; (ii) for residual networks with batch normalization, on the other hand, skip-connections induce power-law rather than exponential behaviour, leading to well-behaved moments and no pathology.
arXiv commentProceedings of ICML 2019 (with contact info updated and formatting issues fixed). Code available at https://github.com/alabatie/moments-dnns
Cited by in corpus (5)
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application
- Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
- Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
- The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
- Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks