Are GANs Created Equal? A Large-Scale Study

Abstract

Generative adversarial networks (GAN) are a powerful subclass of generative models. Despite a very rich research activity leading to numerous interesting GAN algorithms, it is still very hard to assess which algorithm(s) perform better than others. We conduct a neutral, multi-faceted large-scale empirical study on state-of-the art models and evaluation measures. We find that most models can reach similar scores with enough hyperparameter optimization and random restarts. This suggests that improvements can arise from a higher computational budget and tuning more than fundamental algorithmic changes. To overcome some limitations of the current metrics, we also propose several data sets on which precision and recall can be computed. Our experimental results suggest that future GAN research should be based on more systematic and objective evaluation procedures. Finally, we did not find evidence that any of the tested algorithms consistently outperforms the non-saturating GAN introduced in \cite{goodfellow2014generative}.

NIPS'18: Added a section on the limitations of the study and additional empirical results

Cited by in corpus (21)

• EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
• BAGAN: Data Augmentation with Balancing GAN
• Mixed-precision deep learning based on computational memory
• InSituNet: Deep Image Synthesis for Parameter Space Exploration of Ensemble Simulations
• On Accurate Evaluation of GANs for Language Generation
• Conditional Sig-Wasserstein GANs for Time Series Generation
• Flows for simultaneous manifold learning and density estimation
• Reproducibility in Machine Learning for Health
• Statistical Predictions in String Theory and Deep Generative Models
• Sample-Efficient Imitation Learning via Generative Adversarial Nets
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• LOGAN: Membership Inference Attacks Against Generative Models
• Generative Feature Replay For Class-Incremental Learning
• The Devil is in the Decoder: Classification, Regression and GANs
• DeepMasterPrints: Generating MasterPrints for Dictionary Attacks via Latent Variable Evolution
• Scalable Balanced Training of Conditional Generative Adversarial Neural Networks on Image Data
• Manifold-valued Image Generation with Wasserstein Generative Adversarial Nets
• Relational Mimic for Visual Adversarial Imitation Learning
• Inducing Hierarchical Compositional Model by Sparsifying Generator Network
• Dist-GAN: An Improved GAN using Distance Constraints
• Image Manipulation with Perceptual Discriminators