Decorrelated Batch Normalization

Abstract

Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find that PCA whitening causes a problem we call stochastic axis swapping, which is detrimental to learning. We show that ZCA whitening does not suffer from this problem, permitting successful learning. DBN retains the desirable qualities of BN and further improves BN's optimization efficiency and generalization ability. We design comprehensive experiments to show that DBN can improve the performance of BN on multilayer perceptrons and convolutional neural networks. Furthermore, we consistently improve the accuracy of residual networks on CIFAR-10, CIFAR-100, and ImageNet.

Accepted to CVPR 2018. Code available at https://github.com/umich-vl/DecorrelatedBN

Cited by in corpus (27)

• Concept Whitening for Interpretable Image Recognition
• OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer
• Four Things Everyone Should Know to Improve Batch Normalization
• Whitening for Self-Supervised Representation Learning
• Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks
• Backpropagation-Friendly Eigendecomposition
• Revealing the Structure of Deep Neural Networks via Convex Duality
• Iterative Normalization: Beyond Standardization towards Efficient Whitening
• Batch Group Normalization
• Convex Geometry and Duality of Over-parameterized Neural Networks
• Regularizing activations in neural networks via distribution matching with the Wasserstein metric
• Improving STDP-based Visual Feature Learning with Whitening
• Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition
• Channel Equilibrium Networks for Learning Deep Representation
• Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms
• Deep Multi-View Learning via Task-Optimal CCA
• Improving Unsupervised Domain Adaptation by Reducing Bi-level Feature Redundancy
• Approximated Orthonormal Normalisation in Training Neural Networks
• On Batch Orthogonalization Layers
• Feature Whitening via Gradient Transformation for Improved Convergence
• An Investigation into the Stochasticity of Batch Whitening
• Error Autocorrelation Objective Function for Improved System Modeling
• Group Whitening: Balancing Learning Efficiency and Representational Capacity
• Stochastic Whitening Batch Normalization
• Interpretable Face Manipulation Detection via Feature Whitening
• Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation
• An Internal Covariate Shift Bounding Algorithm for Deep Neural Networks by Unitizing Layers' Outputs