Accelerating Training of Deep Neural Networks with a Standardization Loss

Abstract

A significant advance in accelerating neural network training has been the development of normalization methods, permitting the training of deep models both faster and with better accuracy. These advances come with practical challenges: for instance, batch normalization ties the prediction of individual examples with other examples within a batch, resulting in a network that is heavily dependent on batch size. Layer normalization and group normalization are data-dependent and thus must be continually used, even at test-time. To address the issues that arise from using explicit normalization techniques, we propose to replace existing normalization methods with a simple, secondary objective loss that we term a standardization loss. This formulation is flexible and robust across different batch sizes and surprisingly, this secondary objective accelerates learning on the primary training objective. Because it is a training loss, it is simply removed at test-time, and no further effort is needed to maintain normalized activations. We find that a standardization loss accelerates training on both small- and large-scale image classification experiments, works with a variety of architectures, and is largely robust to training across different batch sizes.

arXiv commentTechnical report. Results presented at WiML 2018

References in corpus (13)

• Auto-Encoding Variational Bayes
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• cuDNN: Efficient Primitives for Deep Learning
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• How Does Batch Normalization Help Optimization?
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Recurrent Batch Normalization
• Understanding Batch Normalization
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
• Saturating Auto-Encoders

Cited by in corpus (1)

• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence