Extended Batch Normalization

Abstract

Batch normalization (BN) has become a standard technique for training the modern deep networks. However, its effectiveness diminishes when the batch size becomes smaller, since the batch statistics estimation becomes inaccurate. That hinders batch normalization's usage for 1) training larger model which requires small batches constrained by memory consumption, 2) training on mobile or embedded devices of which the memory resource is limited. In this paper, we propose a simple but effective method, called extended batch normalization (EBN). For NCHW format feature maps, extended batch normalization computes the mean along the (N, H, W) dimensions, as the same as batch normalization, to maintain the advantage of batch normalization. To alleviate the problem caused by small batch size, extended batch normalization computes the standard deviation along the (N, C, H, W) dimensions, thus enlarges the number of samples from which the standard deviation is computed. We compare extended batch normalization with batch normalization and group normalization on the datasets of MNIST, CIFAR-10/100, STL-10, and ImageNet, respectively. The experiments show that extended batch normalization alleviates the problem of batch normalization with small batch size while achieving close performances to batch normalization with large batch size.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Differentiable Learning-to-Normalize via Switchable Normalization
• Norm matters: efficient and accurate normalization schemes in deep networks
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• Recurrent Batch Normalization
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks

Cited by in corpus (5)

• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• flexgrid2vec: Learning Efficient Visual Representations Vectors
• A New Look at Ghost Normalization
• Signature-Graph Networks
• GCCN: Global Context Convolutional Network