Toward a Generalization Metric for Deep Generative Models

Abstract

Measuring the generalization capacity of Deep Generative Models (DGMs) is difficult because of the curse of dimensionality. Evaluation metrics for DGMs such as Inception Score, Fréchet Inception Distance, Precision-Recall, and Neural Net Divergence try to estimate the distance between the generated distribution and the target distribution using a polynomial number of samples. These metrics are the target of researchers when designing new models. Despite the claims, it is still unclear how well can they measure the generalization capacity of a generative model. In this paper, we investigate the capacity of these metrics in measuring the generalization capacity. We introduce a framework for comparing the robustness of evaluation metrics. We show that better scores in these metrics do not imply better generalization. They can be fooled easily by a generator that memorizes a small subset of the training set. We propose a fix to the NND metric to make it more robust to noise in the generated data. Toward building a robust metric for generalization, we propose to apply the Minimum Description Length principle to the problem of evaluating DGMs. We develop an efficient method for estimating the complexity of Generative Latent Variable Models (GLVMs). Experimental results show that our metric can effectively detect training set memorization and distinguish GLVMs of different generalization capacities. Source code is available at https://github.com/htt210/GeneralizationMetricGAN.

arXiv comment1st I Can't Believe It's Not Better Workshop (ICBINB@NeurIPS 2020). Source code is available at https://github.com/htt210/GeneralizationMetricGAN

References in corpus (12)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Which Training Methods for GANs do actually Converge?
• Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs
• A tutorial introduction to the minimum description length principle
• A Note on the Inception Score
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• Assessing Generative Models via Precision and Recall
• Improving Generalization and Stability of Generative Adversarial Networks
• Geometry Score: A Method For Comparing Generative Adversarial Networks
• Towards GAN Benchmarks Which Require Generalization
• On Catastrophic Forgetting and Mode Collapse in Generative Adversarial Networks
• A Non-Parametric Test to Detect Data-Copying in Generative Models