Online Normalization for Training Neural Networks

Abstract

Online Normalization is a new technique for normalizing the hidden activations of a neural network. Like Batch Normalization, it normalizes the sample dimension. While Online Normalization does not use batches, it is as accurate as Batch Normalization. We resolve a theoretical limitation of Batch Normalization by introducing an unbiased technique for computing the gradient of normalized activations. Online Normalization works with automatic differentiation by adding statistical normalization as a primitive. This technique can be used in cases not covered by some other normalizers, such as recurrent networks, fully connected networks, and networks with activation memory requirements prohibitive for batching. We show its applications to image classification, image segmentation, and language modeling. We present formal proofs and experimental results on ImageNet, CIFAR, and PTB datasets.

arXiv commentPublished at the Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. Code: https://github.com/Cerebras/online-normalization

References in corpus (13)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• One weird trick for parallelizing convolutional neural networks
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Layer Normalization
• 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• L2 Regularization versus Batch and Weight Normalization
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
• Mode Normalization

Cited by in corpus (9)

• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• Review: Deep Learning in Electron Microscopy
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
• Batch norm with entropic regularization turns deterministic autoencoders into generative models
• On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
• Deep Unitary Convolutional Neural Networks