Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
arXiv:2002.10801
Abstract
Conditioning analysis uncovers the landscape of an optimization objective by exploring the spectrum of its curvature matrix. This has been well explored theoretically for linear models. We extend this analysis to deep neural networks (DNNs) in order to investigate their learning dynamics. To this end, we propose layer-wise conditioning analysis, which explores the optimization landscape with respect to each layer independently. Such an analysis is theoretically supported under mild assumptions that approximately hold in practice. Based on our analysis, we show that batch normalization (BN) can stabilize the training, but sometimes result in the false impression of a local minimum, which has detrimental effects on the learning. Besides, we experimentally observe that BN can improve the layer-wise conditioning of the optimization problem. Finally, we find that the last linear layer of a very deep residual network displays ill-conditioned behavior. We solve this problem by only adding one BN layer before the last linear layer, which achieves improved performance over the original and pre-activation residual networks.
arXiv commentAccepted to ECCV 2020. The code is available at: https://github.com/huangleiBuaa/LayerwiseCA
References in corpus (13)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- ADADELTA: An Adaptive Learning Rate Method
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
- Norm matters: efficient and accurate normalization schemes in deep networks
- Natural Neural Networks
- Distributed optimization of deeply nested systems
- A Mean Field Theory of Batch Normalization
- Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
- Estimating the Hessian by Back-propagating Curvature