Assessing Generative Models via Precision and Recall

Abstract

Recent advances in generative modeling have led to an increased interest in the study of statistical divergences as means of model comparison. Commonly used evaluation methods, such as the Frechet Inception Distance (FID), correlate well with the perceived quality of samples and are sensitive to mode dropping. However, these metrics are unable to distinguish between different failure cases since they only yield one-dimensional scores. We propose a novel definition of precision and recall for distributions which disentangles the divergence into two separate dimensions. The proposed notion is intuitive, retains desirable properties, and naturally leads to an efficient algorithm that can be used to evaluate generative models. We relate this notion to total variation as well as to recent evaluation metrics such as Inception Score and FID. To demonstrate the practical utility of the proposed approach we perform an empirical study on several variants of Generative Adversarial Networks and Variational Autoencoders. In an extensive set of experiments we show that the proposed metric is able to disentangle the quality of generated samples from the coverage of the target distribution.

arXiv commentNIPS 2018

Cited by in corpus (63)

• Improved Precision and Recall Metric for Assessing Generative Models
• Adversarial Text-to-Image Synthesis: A Review
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Adversarial Video Generation on Complex Datasets
• Classification Accuracy Score for Conditional Generative Models
• Improved Techniques for Training Score-Based Generative Models
• Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
• Lifelong Generative Modeling
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• High-Fidelity Image Generation With Fewer Labels
• On Self Modulation for Generative Adversarial Networks
• You Only Need Adversarial Supervision for Semantic Image Synthesis
• Reliable Fidelity and Diversity Metrics for Generative Models
• Master your Metrics with Calibration
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• A Large-Scale Study on Regularization and Normalization in GANs
• Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models
• Generative Latent Flow
• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
• A Non-Parametric Test to Detect Data-Copying in Generative Models
• This Person (Probably) Exists. Identity Membership Attacks Against GAN Generated Faces
• Generating unseen complex scenes: are we there yet?
• Learning disconnected manifolds: a no GANs land
• Flow Models for Arbitrary Conditional Likelihoods
• Regularized Autoencoders via Relaxed Injective Probability Flow
• Semantic Bottleneck Scene Generation
• RPGAN: GANs Interpretability via Random Routing
• MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators
• Self-Supervised GAN to Counter Forgetting
• Self-Supervised GANs via Auxiliary Rotation Loss
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• On Characterizing GAN Convergence Through Proximal Duality Gap
• Bounds all around: training energy-based models with bidirectional bounds
• Minimax Rates of Estimating Approximate Differential Privacy
• GANs Can Play Lottery Tickets Too
• Data-efficient visuomotor policy training using reinforcement learning and generative models
• On the expressivity of bi-Lipschitz normalizing flows
• Assessing Dialogue Systems with Distribution Distances
• Variational Diffusion Autoencoders with Random Walk Sampling
• Image-to-Image Translation with Low Resolution Conditioning
• Adversarial Augmentation for Enhancing Classification of Mammography Images
• Inclusive GAN: Improving Data and Minority Coverage in Generative Models
• An Adjusted Nearest Neighbor Algorithm Maximizing the F-Measure from Imbalanced Data
• Precision-Recall Curves Using Information Divergence Frontiers
• A Forest from the Trees: Generation through Neighborhoods
• When do GANs replicate? On the choice of dataset size
• Improving the Evaluation of Generative Models with Fuzzy Logic
• MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining
• Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
• MaskAAE: Latent space optimization for Adversarial Auto-Encoders
• Adversarial sampling of unknown and high-dimensional conditional distributions
• Unrealistic Feature Suppression for Generative Adversarial Networks
• Teaching a GAN What Not to Learn
• An Assessment of GANs for Identity-related Applications
• To Regularize or Not To Regularize? The Bias Variance Trade-off in Regularized AEs
• Ensembles of Generative Adversarial Networks for Disconnected Data
• Continual Density Ratio Estimation in an Online Setting
• Barcode Method for Generative Model Evaluation driven by Topological Data Analysis
• Toward a Generalization Metric for Deep Generative Models
• Forward Operator Estimation in Generative Models with Kernel Transfer Operators
• Joint Text and Label Generation for Spoken Language Understanding