Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning

Abstract

We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible. Unlike previous approaches, our technique can be applied out of the box to all learning scenarios (e.g., online learning, batch learning, fully-connected, convolutional, feedforward, recurrent and mixed --- recurrent and convolutional) and compare favorably with existing approaches. We also propose Lp Normalization for normalizing by different orders of statistical moments. In particular, L1 normalization is well-performing, simple to implement, fast to compute, more biologically-plausible and thus ideal for GPU or hardware implementations.

References in corpus (5)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
• Learning Functions: When Is Deep Better Than Shallow
• How Important is Weight Symmetry in Backpropagation?

Cited by in corpus (9)

• Review: Deep Learning in Electron Microscopy
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Deep Convolutional Neural Network Design Patterns
• Low-Memory Neural Network Training: A Technical Report
• Online Normalization for Training Neural Networks
• Regularizing activations in neural networks via distribution matching with the Wasserstein metric
• Normalization of Neural Networks using Analytic Variance Propagation
• A scale invariant ranking function for learning-to-rank: a real-world use case