On the Effects of Batch and Weight Normalization in Generative Adversarial Networks

Abstract

Generative adversarial networks (GANs) are highly effective unsupervised learning frameworks that can generate very sharp data, even for data such as images with complex, highly multimodal distributions. However GANs are known to be very hard to train, suffering from problems such as mode collapse and disturbing visual artifacts. Batch normalization (BN) techniques have been introduced to address the training. Though BN accelerates the training in the beginning, our experiments show that the use of BN can be unstable and negatively impact the quality of the trained model. The evaluation of BN and numerous other recent schemes for improving GAN training is hindered by the lack of an effective objective quality measure for GAN models. To address these issues, we first introduce a weight normalization (WN) approach for GAN training that significantly improves the stability, efficiency and the quality of the generated samples. To allow a methodical evaluation, we introduce squared Euclidean reconstruction error on a test set as a new objective measure, to assess training performance in terms of speed, stability, and quality of generated samples. Our experiments with a standard DCGAN architecture on commonly used datasets (CelebA, LSUN bedroom, and CIFAR-10) indicate that training using WN is generally superior to BN for GANs, achieving 10% lower mean squared loss for reconstruction and significantly better qualitative results than BN. We further demonstrate the stability of WN on a 21-layer ResNet trained with the CelebA data set. The code for this paper is available at https://github.com/stormraiser/gan-weightnorm-resnet

v3 rejected by NIPS 2017, updated and re-submitted to CVPR 2018. v4: add experiments with ResNet and like to new code

References in corpus (4)

• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Improved Training of Wasserstein GANs
• Energy-based Generative Adversarial Network
• BEGAN: Boundary Equilibrium Generative Adversarial Networks

Cited by in corpus (22)

• Spectral Normalization for Generative Adversarial Networks
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
• A Systematic Survey of Regularization and Normalization in GANs
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Learning to Paint With Model-based Deep Reinforcement Learning
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• On the Evaluation of Conditional GANs
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• A Spectral Regularizer for Unsupervised Disentanglement
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• Normalization of Neural Networks using Analytic Variance Propagation
• Training Deep Neural Networks Without Batch Normalization
• Perceptual Video Super Resolution with Enhanced Temporal Consistency
• TFPnP: Tuning-free Plug-and-Play Proximal Algorithm with Applications to Inverse Imaging Problems
• Backpropagation for Implicit Spectral Densities
• Efficient Super Resolution For Large-Scale Images Using Attentional GAN
• A Domain Agnostic Normalization Layer for Unsupervised Adversarial Domain Adaptation
• Generative Models for Security: Attacks, Defenses, and Opportunities
• NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer
• Learning to Avoid Errors in GANs by Manipulating Input Spaces