Micro-Batch Training with Batch-Channel Normalization and Weight Standardization

Abstract

Batch Normalization (BN) has become an out-of-box technique to improve deep network training. However, its effectiveness is limited for micro-batch training, i.e., each GPU typically has only 1-2 images for training, which is inevitable for many computer vision tasks, e.g., object detection and semantic segmentation, constrained by memory consumption. To address this issue, we propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to bring two success factors of BN into micro-batch training: 1) the smoothing effects on the loss landscape and 2) the ability to avoid harmful elimination singularities along the training trajectory. WS standardizes the weights in convolutional layers to smooth the loss landscape by reducing the Lipschitz constants of the loss and the gradients; BCN combines batch and channel normalizations and leverages estimated statistics of the activations in convolutional layers to keep networks away from elimination singularities. We validate WS and BCN on comprehensive computer vision tasks, including image classification, object detection, instance segmentation, video recognition and semantic segmentation. All experimental results consistently show that WS and BCN improve micro-batch training significantly. Moreover, using WS and BCN with micro-batch training is even able to match or outperform the performances of BN with large-batch training.

References in corpus (16)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Neural Architecture Search with Reinforcement Learning
• Instance Normalization: The Missing Ingredient for Fast Stylization
• The Kinetics Human Action Video Dataset
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• On the importance of single directions for generalization
• Differentiable Learning-to-Normalize via Switchable Normalization
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• Skip Connections Eliminate Singularities
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• A Mean Field Theory of Batch Normalization
• Towards Understanding Regularization in Batch Normalization
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes

Cited by in corpus (39)

• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks
• Graph Barlow Twins: A self-supervised representation learning framework for graphs
• Review: Deep Learning in Electron Microscopy
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
• A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
• Understanding Self-supervised Learning with Dual Deep Networks
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• Scaling Wide Residual Networks for Panoptic Segmentation
• Getting to 99% Accuracy in Interactive Segmentation
• Supervised Transfer Learning at Scale for Medical Imaging
• AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
• Multiplicative update rules for accelerating deep learning training and increasing robustness
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• Rethinking Normalization and Elimination Singularity in Neural Networks
• A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization
• $F$, $B$, Alpha Matting
• Countering Noisy Labels By Learning From Auxiliary Clean Labels
• Learning by Turning: Neural Architecture Aware Optimisation
• Urban Sound Classification : striving towards a fair comparison
• CRNNs for Urban Sound Tagging with spatiotemporal context
• Transfer Learning via Test-Time Neural Networks Aggregation
• All-In-One: Artificial Association Neural Networks
• Investigating and Improving Latent Density Segmentation Models for Aleatoric Uncertainty Quantification in Medical Imaging
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• Towards Learning Universal Audio Representations
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer
• MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization
• A Daily Tourism Demand Prediction Framework Based on Multi-head Attention CNN: The Case of The Foreign Entrant in South Korea
• Batch Normalization with Enhanced Linear Transformation
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Variance Reduction in Deep Learning: More Momentum is All You Need
• ISyNet: Convolutional Neural Networks design for AI accelerator
• Disentangling Transfer and Interference in Multi-Domain Learning
• Breaking the Conventional Forward-Backward Tie in Neural Networks: Activation Functions
• Normalized Convolutional Neural Network