Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise
arXiv:1908.04008
Abstract
Batch Normalization (BN)(Ioffe and Szegedy 2015) normalizes the features of an input image via statistics of a batch of images and hence BN will bring the noise to the gradient of the training loss. Previous works indicate that the noise is important for the optimization and generalization of deep neural networks, but too much noise will harm the performance of networks. In our paper, we offer a new point of view that self-attention mechanism can help to regulate the noise by enhancing instance-specific information to obtain a better regularization effect. Therefore, we propose an attention-based BN called Instance Enhancement Batch Normalization (IEBN) that recalibrates the information of each channel by a simple linear transformation. IEBN has a good capacity of regulating noise and stabilizing network training to improve generalization even in the presence of two kinds of noise attacks during training. Finally, IEBN outperforms BN with only a light parameter increment in image classification tasks for different network structures and benchmark datasets.
References in corpus (12)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- A Learned Representation For Artistic Style
- Modulating early visual processing by language
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Adaptive Gradient Methods with Dynamic Bound of Learning Rate
- Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks
- Differentiable Learning-to-Normalize via Switchable Normalization
- Selective Kernel Networks
- Towards Understanding Regularization in Batch Normalization
- DIANet: Dense-and-Implicit Attention Network