A note on the evaluation of generative models
arXiv:1511.01844
Abstract
Probabilistic generative models can be used for compression, denoising, inpainting, texture synthesis, semi-supervised learning, unsupervised feature learning, and other tasks. Given this wide range of applications, it is not surprising that a lot of heterogeneity exists in the way these models are formulated, trained, and evaluated. As a consequence, direct comparison between models is often difficult. This article reviews mostly known but often underappreciated properties relating to the evaluation and interpretation of generative models with a focus on image models. In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional. Good performance with respect to one criterion therefore need not imply good performance with respect to the other criteria. Our results show that extrapolation from one criterion to another is not warranted and generative models need to be evaluated directly with respect to the application(s) they were intended for. In addition, we provide examples demonstrating that Parzen window estimates should generally be avoided.
References in corpus (5)
Cited by in corpus (141)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Generative Adversarial Networks: An Overview
- End-to-end Optimized Image Compression
- NIPS 2016 Tutorial: Generative Adversarial Networks
- Deep Learning for Single Image Super-Resolution: A Brief Review
- Adversarially Learned Inference
- Autoencoding beyond pixels using a learned similarity metric
- f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
- Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
- Deep Voice: Real-time Neural Text-to-Speech
- Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
- A Review on Deep Learning Techniques for Video Prediction
- Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
- A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
- Variational Approaches for Auto-Encoding Generative Adversarial Networks
- Triple Generative Adversarial Nets
- An empirical study on evaluation metrics of generative adversarial networks
- A Note on the Inception Score
- Training of Quantum Circuits on a Hybrid Quantum Computer
- GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation
- Adversarial Text-to-Image Synthesis: A Review
- How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
- Removal of Batch Effects using Distribution-Matching Residual Networks
- Learning in Implicit Generative Models
- Semantic Object Accuracy for Generative Text-to-Image Synthesis
- Tacotron: Towards End-to-End Speech Synthesis
- GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders
- Generative Adversarial Networks (GANs) in Networking: A Comprehensive Survey & Evaluation
- Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from RGB
- Dual Discriminator Generative Adversarial Nets
- Lifelong Generative Modeling
- Generative Models for Automatic Chemical Design
- Adversarial quantum circuit learning for pure state approximation
- Language GANs Falling Short
- Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
- A review of Generative Adversarial Networks for Electronic Health Records: applications, evaluation measures and data sources
- On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
- Cross-view image synthesis using geometry-guided conditional GANs
- Stabilizing Generative Adversarial Networks: A Survey
- Robust Imitation of Diverse Behaviors
- Maximum Entropy Generators for Energy-Based Models
- Wasserstein Learning of Deep Generative Point Process Models
- CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training
- Implicit Maximum Likelihood Estimation
- Multi-Generator Generative Adversarial Nets
- Generating Multi-Agent Trajectories using Programmatic Weak Supervision
- Activation Maximization Generative Adversarial Nets
- Generating Multi-Categorical Samples with Generative Adversarial Networks
- Deep Energy Estimator Networks
- Comparison of Maximum Likelihood and GAN-based training of Real NVPs
- Ensembles of Generative Adversarial Networks
- Unifying Human and Statistical Evaluation for Natural Language Generation
- BourGAN: Generative Networks with Metric Embeddings
- Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs
- IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
- Learning Hierarchical Features from Generative Models
- Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
- Autoregressive Quantile Networks for Generative Modeling
- On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competition
- Sketch-based Creativity Support Tools using Deep Learning
- Visual Intelligence through Human Interaction
- Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
- Video Imagination from a Single Image with Transformation Generation
- Out-of-Distribution Detection using Multiple Semantic Label Representations
- Perfect density models cannot guarantee anomaly detection
- Generalizing Hamiltonian Monte Carlo with Neural Networks
- Skill Rating for Generative Models
- How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models
- Transferring GANs: generating images from limited data
- Adversarial Attacks on Variational Autoencoders
- eCommerceGAN : A Generative Adversarial Network for E-commerce
- Consensus-based Sequence Training for Video Captioning
- Generative Latent Flow
- ArtGAN: Artwork Synthesis with Conditional Categorical GANs
- Unsupervised Out-of-Distribution Detection with Batch Normalization
- Dynamics Transfer GAN: Generating Video by Transferring Arbitrary Temporal Dynamics from a Source Video to a Single Target Image
- Motion Prediction Under Multimodality with Conditional Stochastic Networks
- DeLiGAN : Generative Adversarial Networks for Diverse and Limited Data
- TraDE: Transformers for Density Estimation
- Teacher-Student Compression with Generative Adversarial Networks
- Low-rank Characteristic Tensor Density Estimation Part I: Foundations
- Learning Residual Images for Face Attribute Manipulation
- Emulating Sunyaev-Zeldovich Images of Galaxy Clusters using Auto-Encoders
- Statistics of Deep Generated Images
- Autoregressive Models: What Are They Good For?
- Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders
- Fast Adaptation in Generative Models with Generative Matching Networks
- A Classifying Variational Autoencoder with Application to Polyphonic Music Generation
- Improved Training with Curriculum GANs
- Invertible DenseNets with Concatenated LipSwish
- Non-parametric estimation of Jensen-Shannon Divergence in Generative Adversarial Network training
- Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
- VAE-Info-cGAN: Generating Synthetic Images by Combining Pixel-level and Feature-level Geospatial Conditional Inputs
- Triple Generative Adversarial Networks
- Flow Models for Arbitrary Conditional Likelihoods
- Regularized Autoencoders via Relaxed Injective Probability Flow
- NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting
- GLOWin: A Flow-based Invertible Generative Framework for Learning Disentangled Feature Representations in Medical Images
- Human Image Generation: A Comprehensive Survey
- Bottleneck Conditional Density Estimation
- Learning to Match Distributions for Domain Adaptation
- Structured Output Learning with Conditional Generative Flows
- Causal Adversarial Network for Learning Conditional and Interventional Distributions
- Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis
- Telescoping Density-Ratio Estimation
- Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment
- Building LEGO Using Deep Generative Models of Graphs
- RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects
- Spatial PixelCNN: Generating Images from Patches
- Image Quality Assessment Techniques Show Improved Training and Evaluation of Autoencoder Generative Adversarial Networks
- An ASR Guided Speech Intelligibility Measure for TTS Model Selection
- Towards Principled Uncertainty Estimation for Deep Neural Networks
- Manifold Topology Divergence: a Framework for Comparing Data Manifolds
- Disentanglement for Discriminative Visual Recognition
- Approximating Human Judgment of Generated Image Quality
- Weakly Supervised Label Learning Flows
- Learning Disentangled Representations of Satellite Image Time Series
- Generative Mixture of Networks
- Evaluating Generative Adversarial Networks on Explicitly Parameterized Distributions
- Holdout-Based Fidelity and Privacy Assessment of Mixed-Type Synthetic Data
- Digits that are not: Generating new types through deep neural nets
- Energy-Based Models with Applications to Speech and Language Processing
- Lesion Conditional Image Generation for Improved Segmentation of Intracranial Hemorrhage from CT Images
- RNN-based Generative Model for Fine-Grained Sketching
- Improving the Evaluation of Generative Models with Fuzzy Logic
- Batch Normalization in the final layer of generative networks
- Generative Adversarial Data Programming
- Updating the generator in PPGN-h with gradients flowing through the encoder
- Ten AI Stepping Stones for Cybersecurity
- MaskAAE: Latent space optimization for Adversarial Auto-Encoders
- Generating Realistic Sequences of Customer-level Transactions for Retail Datasets
- Pixel-wise Conditioning of Generative Adversarial Networks
- Generating new pictures in complex datasets with a simple neural network
- Training Generative Networks with general Optimal Transport distances
- Context-based Image Segment Labeling (CBISL)
- Neighbor Embedding Variational Autoencoder
- A study of traits that affect learnability in GANs
- Conditional Generation of Synthetic Geospatial Images from Pixel-level and Feature-level Inputs
- KGAN: How to Break The Minimax Game in GAN
- A Ladder of Causal Distances
- On the Information Theoretic Distance Measures and Bidirectional Helmholtz Machines