The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks

Abstract

Normalization methods play an important role in enhancing the performance of deep learning while their theoretical understandings have been limited. To theoretically elucidate the effectiveness of normalization, we quantify the geometry of the parameter space determined by the Fisher information matrix (FIM), which also corresponds to the local shape of the loss landscape under certain conditions. We analyze deep neural networks with random initialization, which is known to suffer from a pathologically sharp shape of the landscape when the network becomes sufficiently wide. We reveal that batch normalization in the last layer contributes to drastically decreasing such pathological sharpness if the width and sample number satisfy a specific condition. In contrast, it is hard for batch normalization in the middle hidden layers to alleviate pathological sharpness in many settings. We also found that layer normalization cannot alleviate pathological sharpness either. Thus, we can conclude that batch normalization in the last layer significantly contributes to decreasing the sharpness induced by the FIM.

To appear in NeurIPS 2019

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• On Exact Computation with an Infinitely Wide Neural Net
• Exponential expressivity in deep neural networks through transient chaos
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
• The Emergence of Spectral Universality in Deep Networks

Cited by in corpus (4)

• When Does Preconditioning Help or Hurt Generalization?
• Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks