An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
arXiv:1908.00061
Abstract
Batch normalization has been widely used to improve optimization in deep neural networks. While the uncertainty in batch statistics can act as a regularizer, using these dataset statistics specific to the training set impairs generalization in certain tasks. Recently, alternative methods for normalizing feature activations in neural networks have been proposed. Among them, group normalization has been shown to yield similar, in some domains even superior performance to batch normalization. All these methods utilize a learned affine transformation after the normalization operation to increase representational power. Methods used in conditional computation define the parameters of these transformations as learnable functions of conditioning information. In this work, we study whether and where the conditional formulation of group normalization can improve generalization compared to conditional batch normalization. We evaluate performances on the tasks of visual question answering, few-shot learning, and conditional image generation.
References in corpus (14)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Conditional Generative Adversarial Nets
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Self-Attention Generative Adversarial Networks
- Conditional Image Synthesis With Auxiliary Classifier GANs
- A Learned Representation For Artistic Style
- Modulating early visual processing by language
- A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
- A Note on the Inception Score
- Classification Accuracy Score for Conditional Generative Models
- Batch Normalization is a Cause of Adversarial Vulnerability