Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
arXiv:1702.03275
Abstract
Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.
References in corpus (4)
Cited by in corpus (79)
- Data Augmentation Generative Adversarial Networks
- Regularized Deep Networks in Intelligent Transportation Systems: A Taxonomy and a Case Study
- NiftyNet: a deep-learning platform for medical imaging
- Revisiting Small Batch Training for Deep Neural Networks
- High-Performance Large-Scale Image Recognition Without Normalization
- A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts
- L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
- Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
- Differentiable Learning-to-Normalize via Switchable Normalization
- Towards Learning a Universal Non-Semantic Representation of Speech
- Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks
- Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
- Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks
- Review: Deep Learning in Electron Microscopy
- Male pelvic synthetic CT generation from T1-weighted MRI using 2D and 3D convolutional neural networks
- Rethinking ImageNet Pre-training
- The Non-IID Data Quagmire of Decentralized Machine Learning
- Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
- Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
- EmoNeXt: an Adapted ConvNeXt for Facial Emotion Recognition
- Four Things Everyone Should Know to Improve Batch Normalization
- Improving Electron Micrograph Signal-to-Noise with an Atrous Convolutional Encoder-Decoder
- Investigating Generative Adversarial Networks based Speech Dereverberation for Robust Speech Recognition
- A global method to identify trees outside of closed-canopy forests with medium-resolution satellite imagery
- Low-Memory Neural Network Training: A Technical Report
- Online Continual Learning on Sequences
- Online Normalization for Training Neural Networks
- Concurrent ischemic lesion age estimation and segmentation of CT brain using a Transformer-based network
- TaskNorm: Rethinking Batch Normalization for Meta-Learning
- Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization
- Deep Learning on Image Denoising: An overview
- BeamLearning: an end-to-end Deep Learning approach for the angular localization of sound sources using raw multichannel acoustic pressure data
- Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches
- Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning
- Bridge Damage Detection using a Single-Stage Detector and Field Inspection Images
- LOANet: A Lightweight Network Using Object Attention for Extracting Buildings and Roads from UAV Aerial Remote Sensing Images
- Cross-Iteration Batch Normalization
- PowerNorm: Rethinking Batch Normalization in Transformers
- Have convolutions already made recurrence obsolete for unconstrained handwritten text recognition ?
- UniNeXt: Exploring A Unified Architecture for Vision Recognition
- Riemannian approach to batch normalization
- Switchable Whitening for Deep Representation Learning
- Continual Learning on the Edge with TensorFlow Lite
- A Comprehensive Overhaul of Feature Distillation
- Globally Injective ReLU Networks
- Improving Fingerprint Pore Detection with a Small FCN
- Distributed Training Large-Scale Deep Architectures
- Regularizing by the Variance of the Activations' Sample-Variances
- Rethinking Normalization and Elimination Singularity in Neural Networks
- Normalization of Neural Networks using Analytic Variance Propagation
- Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
- Batch Normalization Preconditioning for Neural Network Training
- Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
- Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
- A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent
- Curriculum By Smoothing
- Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise
- Automatic Dataset Annotation to Learn CNN Pore Description for Fingerprint Recognition
- Batch norm with entropic regularization turns deterministic autoencoders into generative models
- Dense 3D Regression for Hand Pose Estimation
- Accent Estimation of Japanese Words from Their Surfaces and Romanizations for Building Large Vocabulary Accent Dictionaries
- Offset Calibration for Appearance-Based Gaze Estimation via Gaze Decomposition
- Controllable Orthogonalization in Training DNNs
- Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence
- SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition
- CNN-based MultiChannel End-to-End Speech Recognition for everyday home environments
- Generalized Batch Normalization: Towards Accelerating Deep Neural Networks
- Open-Ended Content-Style Recombination Via Leakage Filtering
- Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms
- Leveraging Cross-Utterance Context For ASR Decoding
- GIPSO: Geometrically Informed Propagation for Online Adaptation in 3D LiDAR Segmentation
- Group Whitening: Balancing Learning Efficiency and Representational Capacity
- WeightAlign: Normalizing Activations by Weight Alignment
- Backtracking gradient descent method for general functions, with applications to Deep Learning
- Generating new pictures in complex datasets with a simple neural network
- AdjointBackMapV2: Precise Reconstruction of Arbitrary CNN Unit's Activation via Adjoint Operators
- Multi-scale Convolution Aggregation and Stochastic Feature Reuse for DenseNets
- An Investigation into the Stochasticity of Batch Whitening
- Double Forward Propagation for Memorized Batch Normalization