Classification Accuracy Score for Conditional Generative Models

Abstract

Deep generative models (DGMs) of images are now sufficiently mature that they produce nearly photorealistic samples and obtain scores similar to the data distribution on heuristics such as Frechet Inception Distance (FID). These results, especially on large-scale datasets such as ImageNet, suggest that DGMs are learning the data distribution in a perceptually meaningful space and can be used in downstream tasks. To test this latter hypothesis, we use class-conditional generative models from a number of model classes---variational autoencoders, autoregressive models, and generative adversarial networks (GANs)---to infer the class labels of real data. We perform this inference by training an image classifier using only synthetic data and using the classifier to predict labels on real data. The performance on this task, which we call Classification Accuracy Score (CAS), reveals some surprising results not identified by traditional metrics and constitute our contributions. First, when using a state-of-the-art GAN (BigGAN-deep), Top-1 and Top-5 accuracy decrease by 27.9\% and 41.6\%, respectively, compared to the original data; and conditional generative models from other model classes, such as Vector-Quantized Variational Autoencoder-2 (VQ-VAE-2) and Hierarchical Autoregressive Models (HAMs), substantially outperform GANs on this benchmark. Second, CAS automatically surfaces particular classes for which generative models failed to capture the data distribution, and were previously unknown in the literature. Third, we find traditional GAN metrics such as Inception Score (IS) and FID neither predictive of CAS nor useful when evaluating non-GAN models. Furthermore, in order to facilitate better diagnoses of generative models, we open-source the proposed metric.

References in corpus (22)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Conditional Image Generation with PixelCNN Decoders
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Density estimation using Real NVP
• cGANs with Projection Discriminator
• Improved Precision and Recall Metric for Assessing Generative Models
• Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs
• A Note on the Inception Score
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
• Demystifying MMD GANs
• Assessing Generative Models via Precision and Recall
• Do GANs actually learn the distribution? An empirical study
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models
• On Accurate Evaluation of GANs for Language Generation
• Geometry Score: A Method For Comparing Generative Adversarial Networks
• Autoregressive Quantile Networks for Generative Modeling
• Hierarchical Autoregressive Image Models with Auxiliary Decoders
• Teacher-Student Compression with Generative Adversarial Networks

Cited by in corpus (28)

• Skillful Precipitation Nowcasting using Deep Generative Models of Radar
• Cascaded Diffusion Models for High Fidelity Image Generation
• Adversarial Text-to-Image Synthesis: A Review
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Fixing Data Augmentation to Improve Adversarial Robustness
• LOGAN: Latent Optimisation for Generative Adversarial Networks
• Evaluation Metrics for Conditional Image Generation
• Contrastive Learning with Stronger Augmentations
• Automated curricula through setter-solver interactions
• On the Evaluation of Conditional GANs
• Hierarchical Quantized Autoencoders
• StyleGAN2 Distillation for Feed-forward Image Manipulation
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis
• Synthesizing lesions using contextual GANs improves breast cancer classification on mammograms
• Latent Diffusion Models with Image-Derived Annotations for Enhanced AI-Assisted Cancer Diagnosis in Histopathology
• Image Synthesis From Reconfigurable Layout and Style
• Generative Models as a Data Source for Multiview Representation Learning
• Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?
• Manifold Topology Divergence: a Framework for Comparing Data Manifolds
• Improving Robustness using Generated Data
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
• Ensembling with Deep Generative Views
• Instance Selection for GANs
• Conditional Frechet Inception Distance
• Random Network Distillation as a Diversity Metric for Both Image and Text Generation
• Image Captions are Natural Prompts for Text-to-Image Models