Batch Normalization Orthogonalizes Representations in Deep Random Networks

Abstract

This paper underlines a subtle property of batch-normalization (BN): Successive batch normalizations with random linear transformations make hidden representations increasingly orthogonal across layers of a deep neural network. We establish a non-asymptotic characterization of the interplay between depth, width, and the orthogonality of deep representations. More precisely, under a mild assumption, we prove that the deviation of the representations from orthogonality rapidly decays with depth up to a term inversely proportional to the network width. This result has two main implications: 1) Theoretically, as the depth grows, the distribution of the representation -- after the linear layers -- contracts to a Wasserstein-2 ball around an isotropic Gaussian distribution. Furthermore, the radius of this Wasserstein ball shrinks with the width of the network. 2) In practice, the orthogonality of the representations directly influences the performance of stochastic gradient descent (SGD). When representations are initially aligned, we observe SGD wastes many iterations to orthogonalize representations before the classification. Nevertheless, we experimentally show that starting optimization from orthogonal representations is sufficient to accelerate SGD, with no need for BN.

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Random deep neural networks are biased towards simple functions
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
• Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks

Cited by in corpus (2)

• Convolutional Autoencoder-Based Phase Shift Feedback Compression for Intelligent Reflecting Surface-Assisted Wireless Systems
• Training BatchNorm Only in Neural Architecture Search and Beyond