A note on the evaluation of generative models

Abstract

Probabilistic generative models can be used for compression, denoising, inpainting, texture synthesis, semi-supervised learning, unsupervised feature learning, and other tasks. Given this wide range of applications, it is not surprising that a lot of heterogeneity exists in the way these models are formulated, trained, and evaluated. As a consequence, direct comparison between models is often difficult. This article reviews mostly known but often underappreciated properties relating to the evaluation and interpretation of generative models with a focus on image models. In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional. Good performance with respect to one criterion therefore need not imply good performance with respect to the other criteria. Our results show that extrapolation from one criterion to another is not warranted and generative models need to be evaluated directly with respect to the application(s) they were intended for. In addition, we provide examples demonstrating that Parzen window estimates should generally be avoided.

References in corpus (10)

• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Semi-Supervised Learning with Deep Generative Models
• Texture Synthesis Using Convolutional Neural Networks
• Generative Moment Matching Networks
• Deep Generative Stochastic Networks Trainable by Backprop
• Better Mixing via Deep Representations
• Generative Image Modeling Using Spatial LSTMs
• RNADE: The real-valued neural autoregressive density-estimator
• Minimum Probability Flow Learning
• Variational Generative Stochastic Networks with Collaborative Shaping

Cited by in corpus (142)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Generative Adversarial Networks: An Overview
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• End-to-end Optimized Image Compression
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Deep Learning for Single Image Super-Resolution: A Brief Review
• Adversarially Learned Inference
• Autoencoding beyond pixels using a learned similarity metric
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Deep Voice: Real-time Neural Text-to-Speech
• Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
• A Review on Deep Learning Techniques for Video Prediction
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Triple Generative Adversarial Nets
• An empirical study on evaluation metrics of generative adversarial networks
• A Note on the Inception Score
• Training of Quantum Circuits on a Hybrid Quantum Computer
• GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation
• Adversarial Text-to-Image Synthesis: A Review
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
• Removal of Batch Effects using Distribution-Matching Residual Networks
• Learning in Implicit Generative Models
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Tacotron: Towards End-to-End Speech Synthesis
• GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders
• Generative Adversarial Networks (GANs) in Networking: A Comprehensive Survey & Evaluation
• Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from RGB
• Dual Discriminator Generative Adversarial Nets
• Lifelong Generative Modeling
• Generative Models for Automatic Chemical Design
• Adversarial quantum circuit learning for pure state approximation
• Language GANs Falling Short
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
• A review of Generative Adversarial Networks for Electronic Health Records: applications, evaluation measures and data sources
• On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
• Cross-view image synthesis using geometry-guided conditional GANs
• Stabilizing Generative Adversarial Networks: A Survey
• Robust Imitation of Diverse Behaviors
• Maximum Entropy Generators for Energy-Based Models
• Wasserstein Learning of Deep Generative Point Process Models
• Implicit Maximum Likelihood Estimation
• CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training
• Multi-Generator Generative Adversarial Nets
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Activation Maximization Generative Adversarial Nets
• Generating Multi-Categorical Samples with Generative Adversarial Networks
• Deep Energy Estimator Networks
• Ensembles of Generative Adversarial Networks
• Comparison of Maximum Likelihood and GAN-based training of Real NVPs
• Unifying Human and Statistical Evaluation for Natural Language Generation
• BourGAN: Generative Networks with Metric Embeddings
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs
• Learning Hierarchical Features from Generative Models
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• Autoregressive Quantile Networks for Generative Modeling
• On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competition
• Sketch-based Creativity Support Tools using Deep Learning
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Visual Intelligence through Human Interaction
• Video Imagination from a Single Image with Transformation Generation
• Out-of-Distribution Detection using Multiple Semantic Label Representations
• Perfect density models cannot guarantee anomaly detection
• Generalizing Hamiltonian Monte Carlo with Neural Networks
• Skill Rating for Generative Models
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models
• Transferring GANs: generating images from limited data
• Adversarial Attacks on Variational Autoencoders
• eCommerceGAN : A Generative Adversarial Network for E-commerce
• Generative Latent Flow
• Consensus-based Sequence Training for Video Captioning
• ArtGAN: Artwork Synthesis with Conditional Categorical GANs
• Unsupervised Out-of-Distribution Detection with Batch Normalization
• Dynamics Transfer GAN: Generating Video by Transferring Arbitrary Temporal Dynamics from a Source Video to a Single Target Image
• DeLiGAN : Generative Adversarial Networks for Diverse and Limited Data
• TraDE: Transformers for Density Estimation
• Teacher-Student Compression with Generative Adversarial Networks
• Motion Prediction Under Multimodality with Conditional Stochastic Networks
• Low-rank Characteristic Tensor Density Estimation Part I: Foundations
• Learning Residual Images for Face Attribute Manipulation
• Emulating Sunyaev-Zeldovich Images of Galaxy Clusters using Auto-Encoders
• Statistics of Deep Generated Images
• Autoregressive Models: What Are They Good For?
• Improved Training with Curriculum GANs
• Non-parametric estimation of Jensen-Shannon Divergence in Generative Adversarial Network training
• Fast Adaptation in Generative Models with Generative Matching Networks
• Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders
• Invertible DenseNets with Concatenated LipSwish
• A Classifying Variational Autoencoder with Application to Polyphonic Music Generation
• Flow Models for Arbitrary Conditional Likelihoods
• Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
• VAE-Info-cGAN: Generating Synthetic Images by Combining Pixel-level and Feature-level Geospatial Conditional Inputs
• Regularized Autoencoders via Relaxed Injective Probability Flow
• NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting
• Triple Generative Adversarial Networks
• Human Image Generation: A Comprehensive Survey
• Bottleneck Conditional Density Estimation
• GLOWin: A Flow-based Invertible Generative Framework for Learning Disentangled Feature Representations in Medical Images
• Structured Output Learning with Conditional Generative Flows
• Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis
• Causal Adversarial Network for Learning Conditional and Interventional Distributions
• Learning to Match Distributions for Domain Adaptation
• RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects
• Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment
• Telescoping Density-Ratio Estimation
• Image Quality Assessment Techniques Show Improved Training and Evaluation of Autoencoder Generative Adversarial Networks
• Building LEGO Using Deep Generative Models of Graphs
• An ASR Guided Speech Intelligibility Measure for TTS Model Selection
• Spatial PixelCNN: Generating Images from Patches
• Approximating Human Judgment of Generated Image Quality
• Disentanglement for Discriminative Visual Recognition
• Towards Principled Uncertainty Estimation for Deep Neural Networks
• Weakly Supervised Label Learning Flows
• Manifold Topology Divergence: a Framework for Comparing Data Manifolds
• Learning Disentangled Representations of Satellite Image Time Series
• Evaluating Generative Adversarial Networks on Explicitly Parameterized Distributions
• Digits that are not: Generating new types through deep neural nets
• Energy-Based Models with Applications to Speech and Language Processing
• Holdout-Based Fidelity and Privacy Assessment of Mixed-Type Synthetic Data
• Generative Mixture of Networks
• Lesion Conditional Image Generation for Improved Segmentation of Intracranial Hemorrhage from CT Images
• Generative Adversarial Data Programming
• Improving the Evaluation of Generative Models with Fuzzy Logic
• RNN-based Generative Model for Fine-Grained Sketching
• Batch Normalization in the final layer of generative networks
• MaskAAE: Latent space optimization for Adversarial Auto-Encoders
• Generating Realistic Sequences of Customer-level Transactions for Retail Datasets
• Updating the generator in PPGN-h with gradients flowing through the encoder
• Ten AI Stepping Stones for Cybersecurity
• A study of traits that affect learnability in GANs
• KGAN: How to Break The Minimax Game in GAN
• Context-based Image Segment Labeling (CBISL)
• Conditional Generation of Synthetic Geospatial Images from Pixel-level and Feature-level Inputs
• On the Information Theoretic Distance Measures and Bidirectional Helmholtz Machines
• Pixel-wise Conditioning of Generative Adversarial Networks
• Training Generative Networks with general Optimal Transport distances
• Generating new pictures in complex datasets with a simple neural network
• A Ladder of Causal Distances
• Neighbor Embedding Variational Autoencoder