A Large-Scale Study on Regularization and Normalization in GANs

Abstract

Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant number of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of "tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, as well as neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We discuss and evaluate common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.

arXiv commentRevision accepted to ICML'19: More focus on regularization and normalization aspects. Added recent references and promising future directions

References in corpus (13)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• cuDNN: Efficient Primitives for Deep Learning
• Which Training Methods for GANs do actually Converge?
• Improved Precision and Recall Metric for Assessing Generative Models
• On Convergence and Stability of GANs
• Assessing Generative Models via Precision and Recall
• High-Fidelity Image Generation With Fewer Labels
• On Self Modulation for Generative Adversarial Networks
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling

Cited by in corpus (6)

• Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Model
• A Generative Deep Learning Approach to Stochastic Downscaling of Precipitation Forecasts
• A Systematic Survey of Regularization and Normalization in GANs
• Investigating Object Compositionality in Generative Adversarial Networks
• Deep Learning in Protein Structural Modeling and Design
• Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts