A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks

Abstract

In recent years, plenty of metrics have been proposed to identify networks that are free of gradient explosion and vanishing. However, due to the diversity of network components and complex serial-parallel hybrid connections in modern DNNs, the evaluation of existing metrics usually requires strong assumptions, complex statistical analysis, or has limited application fields, which constraints their spread in the community. In this paper, inspired by the Gradient Norm Equality and dynamical isometry, we first propose a novel metric called Block Dynamical Isometry, which measures the change of gradient norm in individual block. Because our Block Dynamical Isometry is norm-based, its evaluation needs weaker assumptions compared with the original dynamical isometry. To mitigate the challenging derivation, we propose a highly modularized statistical framework based on free probability. Our framework includes several key theorems to handle complex serial-parallel hybrid connections and a library to cover the diversity of network components. Besides, several sufficient prerequisites are provided. Powered by our metric and framework, we analyze extensive initialization, normalization, and network structures. We find that Gradient Norm Equality is a universal philosophy behind them. Then, we improve some existing methods based on our analysis, including an activation function selection strategy for initialization techniques, a new configuration for weight normalization, and a depth-aware way to derive coefficients in SeLU. Moreover, we propose a novel normalization technique named second moment normalization, which is theoretically 30% faster than batch normalization without accuracy loss. Last but not least, our conclusions and methods are evidenced by extensive experiments on multiple models over CIFAR10 and ImageNet.

arXiv commentUnder review as a regular paper in TPAMI

References in corpus (25)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• mixup: Beyond Empirical Risk Minimization
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Stable Architectures for Deep Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• All you need is a good init
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Fixup Initialization: Residual Learning Without Normalization
• The Emergence of Spectral Universality in Deep Networks
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• A Quantization-Friendly Separable Convolution for MobileNets
• Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• A Mean Field Theory of Batch Normalization
• Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function
• The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
• Information Geometry of Orthogonal Initializations and Training
• Spectrum concentration in deep residual learning: a free probability approach
• Partial freeness of random matrices