Group Whitening: Balancing Learning Efficiency and Representational Capacity

Abstract

Batch normalization (BN) is an important technique commonly incorporated into deep learning models to perform standardization within mini-batches. The merits of BN in improving a model's learning efficiency can be further amplified by applying whitening, while its drawbacks in estimating population statistics for inference can be avoided through group normalization (GN). This paper proposes group whitening (GW), which exploits the advantages of the whitening operation and avoids the disadvantages of normalization within mini-batches. In addition, we analyze the constraints imposed on features by normalization, and show how the batch size (group number) affects the performance of batch (group) normalized networks, from the perspective of model's representational capacity. This analysis provides theoretical guidance for applying GW in practice. Finally, we apply the proposed GW to ResNet and ResNeXt architectures and conduct experiments on the ImageNet and COCO benchmarks. Results show that GW consistently improves the performance of different architectures, with absolute gains of $1.02\%$ $\sim$ $1.49\%$ in top-1 accuracy on ImageNet and $1.82\%$ $\sim$ $3.21\%$ in bounding box AP on COCO.

arXiv commentV4: camera version of CVPR 2021. Code available at: https://github.com/huangleiBuaa/GroupWhitening

References in corpus (19)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• On the Number of Linear Regions of Deep Neural Networks
• Optimal whitening and decorrelation
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Understanding and Improving Layer Normalization
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Differentiable Learning-to-Normalize via Switchable Normalization
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Norm matters: efficient and accurate normalization schemes in deep networks
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• Towards Understanding Regularization in Batch Normalization
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• Decorrelated Batch Normalization
• On the Number of Linear Regions of Convolutional Neural Networks
• Whitening and Coloring batch transform for GANs
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• Positional Normalization