An Improved Evaluation Framework for Generative Adversarial Networks

Abstract

In this paper, we propose an improved quantitative evaluation framework for Generative Adversarial Networks (GANs) on generating domain-specific images, where we improve conventional evaluation methods on two levels: the feature representation and the evaluation metric. Unlike most existing evaluation frameworks which transfer the representation of ImageNet inception model to map images onto the feature space, our framework uses a specialized encoder to acquire fine-grained domain-specific representation. Moreover, for datasets with multiple classes, we propose Class-Aware Frechet Distance (CAFD), which employs a Gaussian mixture model on the feature space to better fit the multi-manifold feature distribution. Experiments and analysis on both the feature level and the image level were conducted to demonstrate improvements of our proposed framework over the recently proposed state-of-the-art FID method. To our best knowledge, we are the first to provide counter examples where FID gives inconsistent results with human judgments. It is shown in the experiments that our framework is able to overcome the shortness of FID and improves robustness. Code will be made available.

arXiv comment21 pages, 9 figures, 8 tables

References in corpus (9)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Energy-based Generative Adversarial Network
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Generative Moment Matching Networks
• Mode Regularized Generative Adversarial Networks
• An empirical study on evaluation metrics of generative adversarial networks
• Robust Convolutional Neural Networks under Adversarial Noise

Cited by in corpus (10)

• Mixed-precision deep learning based on computational memory
• A survey of synthetic data augmentation methods in computer vision
• Multi-domain stain normalization for digital pathology: A cycle-consistent adversarial network for whole slide images
• DALLE-URBAN: Capturing the urban design expertise of large text to image transformers
• Let's Get the FACS Straight -- Reconstructing Obstructed Facial Features
• Influence Estimation for Generative Adversarial Networks
• Exocentric to Egocentric Image Generation via Parallel Generative Adversarial Network
• Using Generative Models for Pediatric wbMRI
• Conditional Frechet Inception Distance
• Multiclass non-Adversarial Image Synthesis, with Application to Classification from Very Small Sample