Improved Precision and Recall Metric for Assessing Generative Models

Abstract

The ability to automatically estimate the quality and coverage of the samples produced by a generative model is a vital requirement for driving algorithm research. We present an evaluation metric that can separately and reliably measure both of these aspects in image generation tasks by forming explicit, non-parametric representations of the manifolds of real and generated data. We demonstrate the effectiveness of our metric in StyleGAN and BigGAN by providing several illustrative examples where existing metrics yield uninformative or contradictory results. Furthermore, we analyze multiple design variants of StyleGAN to better understand the relationships between the model architecture, training methods, and the properties of the resulting sample distribution. In the process, we identify new variants that improve the state-of-the-art. We also perform the first principled analysis of truncation methods and identify an improved method. Finally, we extend our metric to estimate the perceptual quality of individual samples, and use this to study latent space interpolations.

NeurIPS 2019 final version

References in corpus (15)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Pixel Recurrent Neural Networks
• Semi-Supervised Learning with Deep Generative Models
• Which Training Methods for GANs do actually Converge?
• Unrolled Generative Adversarial Networks
• Assessing Generative Models via Precision and Recall
• Do GANs actually learn the distribution? An empirical study
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• Megapixel Size Image Creation using Generative Adversarial Networks
• Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization
• Revisiting Precision and Recall Definition for Generative Model Evaluation

Cited by in corpus (26)

• Diffusion Models Beat GANs on Image Synthesis
• Training Generative Adversarial Networks with Limited Data
• Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
• Adversarial Text-to-Image Synthesis: A Review
• Multi-Sensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery
• Classification Accuracy Score for Conditional Generative Models
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Fast Graph Generation via Spectral Diffusion
• On the Evaluation of Conditional GANs
• A Large-Scale Study on Regularization and Normalization in GANs
• Latent Denoising Diffusion GAN: Faster sampling, Higher image quality
• Some Theoretical Insights into Wasserstein GANs
• Generative Adversarial Transformers
• Latent Diffusion Models with Image-Derived Annotations for Enhanced AI-Assisted Cancer Diagnosis in Histopathology
• Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models
• An Integral Projection-based Semantic Autoencoder for Zero-Shot Learning
• Deep Residual Mixture Models
• Building LEGO Using Deep Generative Models of Graphs
• Approximating Human Judgment of Generated Image Quality
• Content-Aware Preserving Image Generation
• GeomCA: Geometric Evaluation of Data Representations
• On the Theoretical Equivalence of Several Trade-Off Curves Assessing Statistical Proximity
• Enhancing Stability in Training Conditional Generative Adversarial Networks via Selective Data Matching
• Adversarial sampling of unknown and high-dimensional conditional distributions
• Instance-Conditioned GAN
• Cyclic image generation using chaotic dynamics