A Note on the Inception Score

Abstract

Deep generative models are powerful tools that have produced impressive results in recent years. These advances have been for the most part empirically driven, making it essential that we use high quality evaluation metrics. In this paper, we provide new insights into the Inception Score, a recently proposed and widely used evaluation metric for generative models, and demonstrate that it fails to provide useful guidance when comparing models. We discuss both suboptimalities of the metric itself and issues with its application. Finally, we call for researchers to be more systematic and careful when evaluating and comparing generative models, as the advancement of the field depends upon it.

Proc. ICML 2018 Workshop on Theoretical Foundations and Applications of Deep Generative Models

References in corpus (8)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• C-RNN-GAN: Continuous recurrent neural networks with adversarial training
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Mode Regularized Generative Adversarial Networks
• An empirical study on evaluation metrics of generative adversarial networks
• Long Text Generation via Adversarial Training with Leaked Information
• Deep Learning the Physics of Transport Phenomena

Cited by in corpus (105)

• Diffusion Models Beat GANs on Image Synthesis
• How Generative Adversarial Networks and Their Variants Work: An Overview
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• Adversarial Text-to-Image Synthesis: A Review
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Adversarial Video Generation on Complex Datasets
• Classification Accuracy Score for Conditional Generative Models
• Improved Techniques for Training Score-Based Generative Models
• Scalable agent alignment via reward modeling: a research direction
• Enhanced Balancing GAN: Minority-class Image Generation
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• The Unusual Effectiveness of Averaging in GAN Training
• High-Fidelity Image Generation With Fewer Labels
• Fashion-Gen: The Generative Fashion Dataset and Challenge
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Language GANs Falling Short
• On Self Modulation for Generative Adversarial Networks
• QuGAN: A Quantum State Fidelity based Generative Adversarial Network
• Geometry Score: A Method For Comparing Generative Adversarial Networks
• Conditional GAN for timeseries generation
• TimeMatch: Unsupervised Cross-Region Adaptation by Temporal Shift Estimation
• On gradient regularizers for MMD GANs
• DermGAN: Synthetic Generation of Clinical Skin Images with Pathology
• Human Motion Modeling using DVGANs
• Towards Fine-grained Human Pose Transfer with Detail Replenishing Network
• Sampling Strategies for GAN Synthetic Data
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• Lower Dimensional Kernels for Video Discriminators
• Cross-Modal Contrastive Learning for Text-to-Image Generation
• Racial Faces in-the-Wild: Reducing Racial Bias by Information Maximization Adaptation Network
• Conditional Generation of Medical Images via Disentangled Adversarial Inference
• Adversarial Learning with Local Coordinate Coding
• Language Guided Fashion Image Manipulation with Feature-wise Transformations
• Simple yet Effective Way for Improving the Performance of GAN
• On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competition
• Hierarchical Autoregressive Image Models with Auxiliary Decoders
• Generative Adversarial Network Architectures For Image Synthesis Using Capsule Networks
• Text-to-Image-to-Text Translation using Cycle Consistent Adversarial Networks
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models
• Backdoor Attack through Frequency Domain
• A Relaxed Inertial Forward-Backward-Forward Algorithm for Solving Monotone Inclusions with Application to GANs
• Mimicry: Towards the Reproducibility of GAN Research
• Gesture-to-Gesture Translation in the Wild via Category-Independent Conditional Maps
• Finding Mixed Nash Equilibria of Generative Adversarial Networks
• Multi-StyleGAN: Towards Image-Based Simulation of Time-Lapse Live-Cell Microscopy
• TequilaGAN: How to easily identify GAN samples
• Near-Term Quantum-Classical Associative Adversarial Networks
• A Generative Adversarial Approach To ECG Synthesis And Denoising
• Alleviating Mode Collapse in GAN via Diversity Penalty Module
• GILT: Generating Images from Long Text
• Investigating Under and Overfitting in Wasserstein Generative Adversarial Networks
• Improved Training with Curriculum GANs
• How does Lipschitz Regularization Influence GAN Training?
• Metropolis-Hastings view on variational inference and adversarial training
• A Decentralized Adaptive Momentum Method for Solving a Class of Min-Max Optimization Problems
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• Self-Supervised GANs via Auxiliary Rotation Loss
• Robustness Evaluation of Stacked Generative Adversarial Networks using Metamorphic Testing
• Jigsaw-VAE: Towards Balancing Features in Variational Autoencoders
• GANs beyond divergence minimization
• Learning Implicit Generative Models with the Method of Learned Moments
• Out-of-Sample Extrapolation with Neuron Editing
• Synthetic-Neuroscore: Using A Neuro-AI Interface for Evaluating Generative Adversarial Networks
• Scalable Balanced Training of Conditional Generative Adversarial Neural Networks on Image Data
• Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment
• Quality Evaluation of GANs Using Cross Local Intrinsic Dimensionality
• FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery
• Deep Generative Learning via Variational Gradient Flow
• Visual Evaluation of Generative Adversarial Networks for Time Series Data
• Approximating Human Judgment of Generated Image Quality
• Score-Guided Generative Adversarial Networks
• Learning Global and Local Features of Normal Brain Anatomy for Unsupervised Abnormality Detection
• Training Generative Adversarial Networks in One Stage
• Towards Audio to Scene Image Synthesis using Generative Adversarial Network
• On Computation and Generalization of GANs with Spectrum Control
• Guided Generative Adversarial Neural Network for Representation Learning and High Fidelity Audio Generation using Fewer Labelled Audio Data
• Text-to-Image Generation Grounded by Fine-Grained User Attention
• An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
• A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Dataset
• Inferential Wasserstein Generative Adversarial Networks
• Semi-Implicit Generative Model
• Everything's Talkin': Pareidolia Face Reenactment
• How good is my GAN?
• Lipschitz Constrained GANs via Boundedness and Continuity
• Tempered Adversarial Networks
• Lessons Learned from the Training of GANs on Artificial Datasets
• A 3D Mesh-based Lifting-and-Projection Network for Human Pose Transfer
• Prb-GAN: A Probabilistic Framework for GAN Modelling
• Less Memory, Faster Speed: Refining Self-Attention Module for Image Reconstruction
• Hone as You Read: A Practical Type of Interactive Summarization
• Detecting GAN generated errors
• Voice command generation using Progressive Wavegans
• Random Network Distillation as a Diversity Metric for Both Image and Text Generation
• Updating the generator in PPGN-h with gradients flowing through the encoder
• ChainGAN: A sequential approach to GANs
• Stabilizing Training of Generative Adversarial Nets via Langevin Stein Variational Gradient Descent
• The Implicit Metropolis-Hastings Algorithm
• The Impact of Hole Geometry on Relative Robustness of In-Painting Networks: An Empirical Study
• LEGAN: Disentangled Manipulation of Directional Lighting and Facial Expressions by Leveraging Human Perceptual Judgements
• Toward a Generalization Metric for Deep Generative Models
• Neural Network based Explicit Mixture Models and Expectation-maximization based Learning
• Geometric Generalization Based Zero-Shot Learning Dataset Infinite World: Simple Yet Powerful
• Multiclass non-Adversarial Image Synthesis, with Application to Classification from Very Small Sample
• An Empirical Study on GANs with Margin Cosine Loss and Relativistic Discriminator
• BooVAE: Boosting Approach for Continual Learning of VAE