Mode Normalization

Abstract

Normalization methods are a central building block in the deep learning toolbox. They accelerate and stabilize training, while decreasing the dependence on manually tuned learning rate schedules. When learning from multi-modal distributions, the effectiveness of batch normalization (BN), arguably the most prominent normalization method, is reduced. As a remedy, we propose a more flexible approach: by extending the normalization to more than a single mean and variance, we detect modes of data on-the-fly, jointly normalizing samples that share common features. We demonstrate that our method outperforms BN and other widely used normalization techniques in several experiments, including single and multi-task datasets.

Cited by in corpus (6)

• Four Things Everyone Should Know to Improve Batch Normalization
• Online Normalization for Training Neural Networks
• Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift
• Regularizing activations in neural networks via distribution matching with the Wasserstein metric
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Towards Defending Multiple ${\ell }_p$-norm Bounded Adversarial Perturbations via Gated Batch Normalization