Generating Diverse High-Fidelity Images with VQ-VAE-2

Abstract

We explore the use of Vector Quantized Variational AutoEncoder (VQ-VAE) models for large scale image generation. To this end, we scale and enhance the autoregressive priors used in VQ-VAE to generate synthetic samples of much higher coherence and fidelity than possible before. We use simple feed-forward encoder and decoder networks, making our model an attractive candidate for applications where the encoding and/or decoding speed is critical. Additionally, VQ-VAE requires sampling an autoregressive model only in the compressed latent space, which is an order of magnitude faster than sampling in the pixel space, especially for large images. We demonstrate that a multi-scale hierarchical organization of VQ-VAE, augmented with powerful priors over the latent codes, is able to generate samples with quality that rivals that of state of the art Generative Adversarial Networks on multifaceted datasets such as ImageNet, while not suffering from GAN's known shortcomings such as mode collapse and lack of diversity.

References in corpus (20)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Unsupervised Cross-Domain Image Generation
• Improved Precision and Recall Metric for Assessing Generative Models
• Variational Lossy Autoencoder
• A Note on the Inception Score
• Assessing Generative Models via Precision and Recall
• Classification Accuracy Score for Conditional Generative Models
• The challenge of realistic music generation: modelling raw audio at scale
• Generative Image Modeling Using Spatial LSTMs
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
• PixelSNAIL: An Improved Autoregressive Generative Model
• Discriminator Rejection Sampling
• Hierarchical Autoregressive Image Models with Auxiliary Decoders

Cited by in corpus (36)

• Diffusion Models Beat GANs on Image Synthesis
• Zero-Shot Text-to-Image Generation
• Adversarial Text-to-Image Synthesis: A Review
• Adversarial Video Generation on Complex Datasets
• Classification Accuracy Score for Conditional Generative Models
• Lifelong Generative Modeling
• From Variational to Deterministic Autoencoders
• Vector-quantized Image Modeling with Improved VQGAN
• Controlling generative models with continuous factors of variations
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• Long-Short Transformer: Efficient Transformers for Language and Vision
• Robust Training of Vector Quantized Bottleneck Models
• Cross-Camera Feature Prediction for Intra-Camera Supervised Person Re-identification across Distant Scenes
• On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competition
• A unified framework for 21cm tomography sample generation and parameter inference with Progressively Growing GANs
• Non Gaussian Denoising Diffusion Models
• A Generative Model of Galactic Dust Emission Using Variational Inference
• VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
• Bilateral Denoising Diffusion Models
• Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
• Neural Design Network: Graphic Layout Generation with Constraints
• Demystifying Inter-Class Disentanglement
• High- and Low-level image component decomposition using VAEs for improved reconstruction and anomaly detection
• Unsupervised Eyeglasses Removal in the Wild
• VQ-DRAW: A Sequential Discrete VAE
• The Multi-speaker Multi-style Voice Cloning Challenge 2021
• Augmentation-Interpolative AutoEncoders for Unsupervised Few-Shot Image Generation
• Classify and Generate: Using Classification Latent Space Representations for Image Generations
• Lessons Learned from the Training of GANs on Artificial Datasets
• Set Distribution Networks: a Generative Model for Sets of Images
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay
• NP-DRAW: A Non-Parametric Structured Latent Variable Model for Image Generation
• Discrete Variational Attention Models for Language Generation
• Zero-Shot Translation using Diffusion Models