Towards GAN Benchmarks Which Require Generalization

Abstract

For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic. We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model. In search of such a metric, we turn to neural network divergences (NNDs), which are defined in terms of a neural network trained to distinguish between distributions. The resulting benchmarks cannot be "won" by training set memorization, while still being perceptually correlated and computable only from samples. We survey past work on using NNDs for evaluation and implement an example black-box metric based on these ideas. Through experimental validation we show that it can effectively measure diversity, sample quality, and generalization.

ICLR 2019 conference paper

Cited by in corpus (17)

• Deep learning for molecular design - a review of the state of the art
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• A review of radar-based nowcasting of precipitation and applicable machine learning techniques
• Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
• A Non-Parametric Test to Detect Data-Copying in Generative Models
• Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples
• Fair Generative Modeling via Weak Supervision
• Understanding Overparameterization in Generative Adversarial Networks
• Regularizing Generative Adversarial Networks under Limited Data
• How does Lipschitz Regularization Influence GAN Training?
• Compositional Fine-Grained Low-Shot Learning
• A Mathematical Framework for Learning Probability Distributions
• Stable Rank Normalization for Improved Generalization in Neural Networks and GANs
• Optimal 1-Wasserstein Distance for WGANs
• Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution
• Toward a Generalization Metric for Deep Generative Models
• Identifying and Exploiting Structures for Reliable Deep Learning