Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
arXiv:2009.02773
Abstract
Spectral normalization (SN) is a widely-used technique for improving the stability and sample quality of Generative Adversarial Networks (GANs). However, there is currently limited understanding of why SN is effective. In this work, we show that SN controls two important failure modes of GAN training: exploding and vanishing gradients. Our proofs illustrate a (perhaps unintentional) connection with the successful LeCun initialization. This connection helps to explain why the most popular implementation of SN for GANs requires no hyper-parameter tuning, whereas stricter implementations of SN have poor empirical performance out-of-the-box. Unlike LeCun initialization which only controls gradient vanishing at the beginning of training, SN preserves this property throughout training. Building on this theoretical understanding, we propose a new spectral normalization technique: Bidirectional Scaled Spectral Normalization (BSSN), which incorporates insights from later improvements to LeCun initialization: Xavier initialization and Kaiming initialization. Theoretically, we show that BSSN gives better gradient control than SN. Empirically, we demonstrate that it outperforms SN in sample quality and training stability on several benchmark datasets.
arXiv comment54 pages, 74 figures
References in corpus (13)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- On the difficulty of training Recurrent Neural Networks
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- The relativistic discriminator: a key element missing from standard GAN
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Spectral Norm Regularization for Improving the Generalizability of Deep Learning
- Stochastic Adversarial Video Prediction
- Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
- Generalizable Adversarial Training via Spectral Normalization
- On the distance between two neural networks and the stability of learning