An Investigation into the Stochasticity of Batch Whitening

Abstract

Batch Normalization (BN) is extensively employed in various network architectures by performing standardization within mini-batches. A full understanding of the process has been a central target in the deep learning communities. Unlike existing works, which usually only analyze the standardization operation, this paper investigates the more general Batch Whitening (BW). Our work originates from the observation that while various whitening transformations equivalently improve the conditioning, they show significantly different behaviors in discriminative scenarios and training Generative Adversarial Networks (GANs). We attribute this phenomenon to the stochasticity that BW introduces. We quantitatively investigate the stochasticity of different whitening transformations and show that it correlates well with the optimization behaviors during training. We also investigate how stochasticity relates to the estimation of population statistics during inference. Based on our analysis, we provide a framework for designing and comparing BW algorithms in different scenarios. Our proposed BW algorithm improves the residual networks by a significant margin on ImageNet classification. Besides, we show that the stochasticity of BW can improve the GAN's performance with, however, the sacrifice of the training stability.

arXiv commentAccepted to CVPR 2020. The Code is available at https://github.com/huangleiBuaa/StochasticityBW

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Layer Normalization
• Optimal whitening and decorrelation
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Differentiable Learning-to-Normalize via Switchable Normalization
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Norm matters: efficient and accurate normalization schemes in deep networks
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• Decorrelated Batch Normalization
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
• Whitening and Coloring batch transform for GANs