On the Evaluation of Conditional GANs

Abstract

Conditional Generative Adversarial Networks (cGANs) are finding increasingly widespread use in many application domains. Despite outstanding progress, quantitative evaluation of such models often involves multiple distinct metrics to assess different desirable properties, such as image quality, conditional consistency, and intra-conditioning diversity. In this setting, model benchmarking becomes a challenge, as each metric may indicate a different "best" model. In this paper, we propose the Frechet Joint Distance (FJD), which is defined as the Frechet distance between joint distributions of images and conditioning, allowing it to implicitly capture the aforementioned properties in a single metric. We conduct proof-of-concept experiments on a controllable synthetic dataset, which consistently highlight the benefits of FJD when compared to currently established metrics. Moreover, we use the newly introduced metric to compare existing cGAN-based models for a variety of conditioning modalities (e.g. class labels, object masks, bounding boxes, images, and text captions). We show that FJD can be used as a promising single metric for cGAN benchmarking and model selection. Code can be found at https://github.com/facebookresearch/fjd.

References in corpus (21)

• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• WaveNet: A Generative Model for Raw Audio
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Toward Multimodal Image-to-Image Translation
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Improved Precision and Recall Metric for Assessing Generative Models
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
• Classification Accuracy Score for Conditional Generative Models
• Diversity-Sensitive Conditional Generative Adversarial Networks
• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• ChatPainter: Improving Text to Image Generation using Dialogue
• On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
• Gang of GANs: Generative Adversarial Networks with Maximum Margin Ranking

Cited by in corpus (13)

• Adversarial Text-to-Image Synthesis: A Review
• Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
• Data synthesis and adversarial networks: A review and meta-analysis in cancer imaging
• Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression
• Machine learning approaches for automatic defect detection in photovoltaic systems
• Generating unseen complex scenes: are we there yet?
• Continuous Conditional Generative Adversarial Networks: Novel Empirical Losses and Label Input Mechanisms
• Semi-Supervised Learning of Mutually Accelerated MRI Synthesis without Fully-Sampled Ground Truths
• Conditional Frechet Inception Distance
• Multiple Generative Adversarial Networks Analysis for Predicting Photographers' Retouching
• Partially Conditioned Generative Adversarial Networks
• Generating Multi-type Temporal Sequences to Mitigate Class-imbalanced Problem
• Establishing an Evaluation Metric to Quantify Climate Change Image Realism