An empirical study on evaluation metrics of generative adversarial networks

Abstract

Evaluating generative adversarial networks (GANs) is inherently challenging. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the problem of how to evaluate the evaluation metrics. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. With a series of carefully designed experiments, we comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbor (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far they are from learning the target distribution.

arXiv commentarXiv admin note: text overlap with arXiv:1802.03446 by other authors

References in corpus (17)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Improved Training of Wasserstein GANs
• Going Deeper with Convolutions
• Improved Techniques for Training GANs
• Understanding deep learning requires rethinking generalization
• Energy-based Generative Adversarial Network
• Understanding Deep Convolutional Networks
• Autoencoding beyond pixels using a learned similarity metric
• Semi-Supervised Learning with Generative Adversarial Networks
• A note on the evaluation of generative models
• Mode Regularized Generative Adversarial Networks
• Least Squares Generative Adversarial Networks
• Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities
• A Test of Relative Similarity For Model Selection in Generative Models

Cited by in corpus (44)

• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Privacy-preserving Spatiotemporal Scenario Generation of Renewable Energies: A Federated Deep Generative Learning Approach
• A Note on the Inception Score
• Using GANs for Sharing Networked Time Series Data: Challenges, Initial Promise, and Open Questions
• A Deep-Learning Intelligent System Incorporating Data Augmentation for Short-Term Voltage Stability Assessment of Power Systems
• Deep learning based on Transformer architecture for power system short-term voltage stability assessment with class imbalance
• Differentially Private Synthetic Medical Data Generation using Convolutional GANs
• Generative Adversarial Networks (GANs) in Networking: A Comprehensive Survey & Evaluation
• Multi-Spectral Image Synthesis for Crop/Weed Segmentation in Precision Farming
• PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows
• On gradient regularizers for MMD GANs
• An Improved Evaluation Framework for Generative Adversarial Networks
• MO-PaDGAN: Reparameterizing Engineering Designs for Augmented Multi-objective Optimization
• Multi-domain stain normalization for digital pathology: A cycle-consistent adversarial network for whole slide images
• Synthesis of Realistic ECG using Generative Adversarial Networks
• Physics-informed GANs for Coastal Flood Visualization
• On Filter Generalization for Music Bandwidth Extension Using Deep Neural Networks
• Ultra-High-Resolution Detector Simulation with Intra-Event Aware GAN and Self-Supervised Relational Reasoning
• Adversarial Lipschitz Regularization
• A Non-Parametric Test to Detect Data-Copying in Generative Models
• Generating unseen complex scenes: are we there yet?
• PI-REC: Progressive Image Reconstruction Network With Edge and Color Domain
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• Face-to-Parameter Translation for Game Character Auto-Creation
• Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models
• Delving into the Continuous Domain Adaptation
• Two-sample Testing Using Deep Learning
• Synthetic-Neuroscore: Using A Neuro-AI Interface for Evaluating Generative Adversarial Networks
• MatchingGAN: Matching-based Few-shot Image Generation
• Feasibility of Modeling Orthogonal Frequency-Division Multiplexing Communication Signals with Unsupervised Generative Adversarial Networks
• F2GAN: Fusing-and-Filling GAN for Few-shot Image Generation
• Entropy-regularized Optimal Transport Generative Models
• GANs with Variational Entropy Regularizers: Applications in Mitigating the Mode-Collapse Issue
• A Neuro-AI Interface for Evaluating Generative Adversarial Networks
• Air-Writing Translater: A Novel Unsupervised Domain Adaptation Method for Inertia-Trajectory Translation of In-air Handwriting
• GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks
• Benefiting Deep Latent Variable Models via Learning the Prior and Removing Latent Regularization
• Neural Network based Explicit Mixture Models and Expectation-maximization based Learning
• Quantifying point cloud realism through adversarially learned latent representations
• Synthetic Periocular Iris PAI from a Small Set of Near-Infrared-Images
• Restrained Generative Adversarial Network against Overfitting in Numeric Data Augmentation
• Multiclass non-Adversarial Image Synthesis, with Application to Classification from Very Small Sample
• Implicit Generative Copulas