PixelVAE: A Latent Variable Model for Natural Images

Abstract

Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficulty capturing small details. PixelCNN models details very well, but lacks a latent code and is difficult to scale for capturing large structures. We present PixelVAE, a VAE model with an autoregressive decoder based on PixelCNN. Our model requires very few expensive autoregressive layers compared to PixelCNN and learns latent codes that are more compressed than a standard VAE while still capturing most non-trivial structure. Finally, we extend our model to a hierarchy of latent variables at different scales. Our model achieves state-of-the-art performance on binarized MNIST, competitive performance on 64x64 ImageNet, and high-quality samples on the LSUN bedrooms dataset.

Cited by in corpus (29)

• An Introduction to Variational Autoencoders
• Joint Autoregressive and Hierarchical Priors for Learned Image Compression
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• Learning Robust Representations via Multi-View Information Bottleneck
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling
• Conditional Flow Variational Autoencoders for Structured Sequence Prediction
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• The continuous Bernoulli: fixing a pervasive error in variational autoencoders
• Practical Lossless Compression with Latent Variables using Bits Back Coding
• Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition
• UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Decision-Making with Auto-Encoding Variational Bayes
• VFlow: More Expressive Generative Flows with Variational Data Augmentation
• Wavelet Flow: Fast Training of High Resolution Normalizing Flows
• Variational Autoencoders with Normalizing Flow Decoders
• Preventing Posterior Collapse with delta-VAEs
• Maximizing Mutual Information for Tacotron
• On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder
• TwoStreamVAN: Improving Motion Modeling in Video Generation
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• Rate-Regularization and Generalization in VAEs
• Dynamic Topic Language Model on Heterogeneous Children's Mental Health Clinical Notes
• Image Style Transfer and Content-Style Disentanglement
• Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling
• From abstract items to latent spaces to observed data and back: Compositional Variational Auto-Encoder
• Self-Reflective Variational Autoencoder
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory