Cross-Iteration Batch Normalization
arXiv:2002.05712
Abstract
A well-known issue of Batch Normalization is its significantly reduced effectiveness in the case of small mini-batch sizes. When a mini-batch contains few examples, the statistics upon which the normalization is defined cannot be reliably estimated from it during a training iteration. To address this problem, we present Cross-Iteration Batch Normalization (CBN), in which examples from multiple recent iterations are jointly utilized to enhance estimation quality. A challenge of computing statistics over multiple iterations is that the network activations from different iterations are not comparable to each other due to changes in network weights. We thus compensate for the network weight changes via a proposed technique based on Taylor polynomials, so that the statistics can be accurately estimated and batch normalization can be effectively applied. On object detection and image classification with small mini-batch sizes, CBN is found to outperform the original batch normalization and a direct calculation of statistics over previous iterations without the proposed compensation technique. Code is available at https://github.com/Howal/Cross-iterationBatchNorm .
arXiv commentAccepted to CVPR 2021
References in corpus (6)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Differentiable Learning-to-Normalize via Switchable Normalization
- DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices
- SSN: Learning Sparse Switchable Normalization via SparsestMax
Cited by in corpus (5)
- YOLOv4: Optimal Speed and Accuracy of Object Detection
- Aerial Images Processing for Car Detection using Convolutional Neural Networks: Comparison between Faster R-CNN and YoloV3
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application
- Rethinking "Batch" in BatchNorm
- Separable Batch Normalization for Robust Facial Landmark Localization with Cross-protocol Network Training