Reliable Fidelity and Diversity Metrics for Generative Models

Abstract

Devising indicative evaluation metrics for the image generation task remains an open problem. The most widely used metric for measuring the similarity between real and generated images has been the Fréchet Inception Distance (FID) score. Because it does not differentiate the fidelity and diversity aspects of the generated images, recent papers have introduced variants of precision and recall metrics to diagnose those properties separately. In this paper, we show that even the latest version of the precision and recall metrics are not reliable yet. For example, they fail to detect the match between two identical distributions, they are not robust against outliers, and the evaluation hyperparameters are selected arbitrarily. We propose density and coverage metrics that solve the above issues. We analytically and experimentally show that density and coverage provide more interpretable and reliable signals for practitioners than the existing metrics. Code: https://github.com/clovaai/generative-evaluation-prdc.

arXiv commentFirst two authors have contributed equally; ICML 2020 accepted

References in corpus (8)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Better Mixing via Deep Representations
• Assessing Generative Models via Precision and Recall
• NSML: Meet the MLaaS platform with a real-world case study
• Adversarial Audio Synthesis

Cited by in corpus (4)

• Adversarial Text-to-Image Synthesis: A Review
• Neural Language Generation: Formulation, Methods, and Evaluation
• Building LEGO Using Deep Generative Models of Graphs
• Synthesizing Photorealistic Images with Deep Generative Learning