Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes

Abstract

Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural networks. On the other hand, layer normalization normalizes the activations across all activities within a layer. This was shown to work well in the recurrent setting. In this paper we propose a unified view of normalization techniques, as forms of divisive normalization, which includes layer and batch normalization as special cases. Our second contribution is the finding that a small modification to these normalization schemes, in conjunction with a sparse regularizer on the activations, leads to significant benefits over standard normalization techniques. We demonstrate the effectiveness of our unified divisive normalization framework in the context of convolutional neural nets and recurrent neural networks, showing improvements over baselines in image classification, language modeling as well as super-resolution.

Published as a conference paper at ICLR 2017

Cited by in corpus (24)

• Generalisation in humans and deep neural networks
• Neuromorphic Deep Learning Machines
• Differentiable Learning-to-Normalize via Switchable Normalization
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Review: Deep Learning in Electron Microscopy
• Neural Sampling Machine with Stochastic Synapse allows Brain-like Learning and Inference
• Large Associative Memory Problem in Neurobiology and Machine Learning
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• DeepSignals: Predicting Intent of Drivers Through Visual Signals
• Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Why Not to Use Zero Imputation? Correcting Sparsity Bias in Training Neural Networks
• Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding
• Iterative Normalization: Beyond Standardization towards Efficient Whitening
• Extended Batch Normalization
• Batch Group Normalization
• Normalization of Neural Networks using Analytic Variance Propagation
• Bio-Inspired Hashing for Unsupervised Similarity Search
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
• Instance-Level Meta Normalization
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions
• An Investigation into the Stochasticity of Batch Whitening
• Group Whitening: Balancing Learning Efficiency and Representational Capacity