Normalization Techniques in Training DNNs: Methodology, Analysis and Application

Abstract

Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various applications. This paper reviews and comments on the past, present and future of normalization methods in the context of DNN training. We provide a unified picture of the main motivation behind different approaches from the perspective of optimization, and present a taxonomy for understanding the similarities and differences between them. Specifically, we decompose the pipeline of the most representative normalizing activation methods into three components: the normalization area partitioning, normalization operation and normalization representation recovery. In doing so, we provide insight for designing new normalization technique. Finally, we discuss the current progress in understanding normalization methods, and provide a comprehensive review of the applications of normalization for particular tasks, in which it can effectively solve the key issues.

20 pages

References in corpus (80)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• On the difficulty of training Recurrent Neural Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Self-Attention Generative Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• On the Number of Linear Regions of Deep Neural Networks
• A Learned Representation For Artistic Style
• On First-Order Meta-Learning Algorithms
• Large Batch Training of Convolutional Networks
• Optimal whitening and decorrelation
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Designing Neural Network Architectures using Reinforcement Learning
• Revisiting Batch Normalization For Practical Domain Adaptation
• New insights and perspectives on the natural gradient method
• Concept Whitening for Interpretable Image Recognition
• Averaging Weights Leads to Wider Optima and Better Generalization
• L2 Regularization versus Batch and Weight Normalization
• Understanding and Improving Layer Normalization
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• On Layer Normalization in the Transformer Architecture
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Norm matters: efficient and accurate normalization schemes in deep networks
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Krylov Subspace Descent for Deep Learning
• Learning Feature Hierarchies with Centered Deep Boltzmann Machines
• On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
• Sorting out Lipschitz function approximation
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• CapProNet: Deep Feature Learning via Orthogonal Projections onto Capsule Subspaces
• Riemannian batch normalization for SPD neural networks
• WNGrad: Learn the Learning Rate in Gradient Descent
• Evolving Normalization-Activation Layers
• Batch Normalization is a Cause of Adversarial Vulnerability
• Towards Understanding Regularization in Batch Normalization
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• Training Deep Networks with Structured Layers by Matrix Backpropagation
• Generalized BackPropagation, Étude De Cas: Orthogonality
• Optimization on Submanifolds of Convolution Kernels in CNNs
• Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks
• Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization
• Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
• On Feature Normalization and Data Augmentation
• How Does Batch Normalization Help Binary Training?
• Deep Isometric Learning for Visual Recognition
• Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
• DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation
• Towards an Adversarially Robust Normalization Approach
• Cross-Iteration Batch Normalization
• Projection Based Weight Normalization for Deep Neural Networks
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
• Low-Precision Batch-Normalized Activations
• Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
• Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks
• Normalization of Neural Networks using Analytic Variance Propagation
• Characterizing Well-Behaved vs. Pathological Deep Neural Networks
• Regularizing by the Variance of the Activations' Sample-Variances
• Rethinking Normalization and Elimination Singularity in Neural Networks
• Regularizing activations in neural networks via distribution matching with the Wasserstein metric
• Be Like Water: Robustness to Extraneous Variables Via Adaptive Feature Normalization
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
• Towards Understanding Normalization in Neural ODEs
• Deep Learning for Inverse Problems: Bounds and Regularizers
• Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction
• A New Look at Ghost Normalization
• IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization
• Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
• Orthogonal Wasserstein GANs
• Large Batch Training Does Not Need Warmup
• Unpaired Image Translation via Adaptive Convolution-based Normalization
• An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
• Finet: Using Fine-grained Batch Normalization to Train Light-weight Neural Networks
• Separating the Effects of Batch Normalization on CNN Training Speed and Stability Using Classical Adaptive Filter Theory

Cited by in corpus (7)

• High-Performance Large-Scale Image Recognition Without Normalization
• Characterizing signal propagation to close the performance gap in unnormalized ResNets
• Deep learning fluid flow reconstruction around arbitrary two-dimensional objects from sparse sensors using conformal mappings
• Deep Neural Networks for Estimation of Gamma-Ray Burst Redshifts
• Self-Contrastive Forward-Forward Algorithm
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Logit Attenuating Weight Normalization