WeightAlign: Normalizing Activations by Weight Alignment
arXiv:2010.07160
Abstract
Batch normalization (BN) allows training very deep networks by normalizing activations by mini-batch sample statistics which renders BN unstable for small batch sizes. Current small-batch solutions such as Instance Norm, Layer Norm, and Group Norm use channel statistics which can be computed even for a single sample. Such methods are less stable than BN as they critically depend on the statistics of a single input sample. To address this problem, we propose a normalization of activation without sample statistics. We present WeightAlign: a method that normalizes the weights by the mean and scaled standard derivation computed within a filter, which normalizes activations without computing any sample statistics. Our proposed method is independent of batch size and stable over a wide range of batch sizes. Because weight statistics are orthogonal to sample statistics, we can directly combine WeightAlign with any method for activation normalization. We experimentally demonstrate these benefits for classification on CIFAR-10, CIFAR-100, ImageNet, for semantic segmentation on PASCAL VOC 2012 and for domain adaptation on Office-31.
arXiv commentThe first three authors contributed equally; accepted by ICPR 2020
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- On the difficulty of training Recurrent Neural Networks
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Revisiting Batch Normalization For Practical Domain Adaptation
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- All you need is a good init
- Fixup Initialization: Residual Learning Without Normalization
- Principled Weight Initialization for Hypernetworks
- Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization