NVAE: A Deep Hierarchical Variational Autoencoder

Abstract

Normalizing flows, autoregressive models, variational autoencoders (VAEs), and deep energy-based models are among competing likelihood-based frameworks for deep generative learning. Among them, VAEs have the advantage of fast and tractable sampling and easy-to-access encoding networks. However, they are currently outperformed by other models such as normalizing flows and autoregressive models. While the majority of the research in VAEs is focused on the statistical challenges, we explore the orthogonal direction of carefully designing neural architectures for hierarchical VAEs. We propose Nouveau VAE (NVAE), a deep hierarchical VAE built for image generation using depth-wise separable convolutions and batch normalization. NVAE is equipped with a residual parameterization of Normal distributions and its training is stabilized by spectral regularization. We show that NVAE achieves state-of-the-art results among non-autoregressive likelihood-based models on the MNIST, CIFAR-10, CelebA 64, and CelebA HQ datasets and it provides a strong baseline on FFHQ. For example, on CIFAR-10, NVAE pushes the state-of-the-art from 2.98 to 2.91 bits per dimension, and it produces high-quality images on CelebA HQ. To the best of our knowledge, NVAE is the first successful VAE applied to natural images as large as 256$\times$256 pixels. The source code is available at https://github.com/NVlabs/NVAE .

arXiv commentNeural Information Processing Systems (NeurIPS) 2020 (spotlight)

References in corpus (37)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Categorical Reparameterization with Gumbel-Softmax
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Opening the Black Box of Deep Neural Networks via Information
• Searching for Activation Functions
• Density estimation using Real NVP
• Training Deep Nets with Sublinear Memory Cost
• Auxiliary Deep Generative Models
• Image Transformer
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Residual Flows for Invertible Generative Modeling
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• The challenge of realistic music generation: modelling raw audio at scale
• Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference
• Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
• Rényi Divergence Variational Inference
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
• Discrete Variational Autoencoders
• DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
• On the Quantitative Analysis of Decoder-Based Generative Models
• DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• VFlow: More Expressive Generative Flows with Variational Data Augmentation
• Bidirectional Helmholtz Machines
• PixelVAE++: Improved PixelVAE with Discrete Prior
• MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
• Undirected Graphical Models as Approximate Posteriors

Cited by in corpus (83)

• Diffusion Models Beat GANs on Image Synthesis
• Improved Denoising Diffusion Probabilistic Models
• Variational Diffusion Models
• Dynamical Variational Autoencoders: A Comprehensive Review
• VideoGPT: Video Generation using VQ-VAE and Transformers
• Vector-quantized Image Modeling with Improved VQGAN
• Denoising diffusion algorithm for inverse design of microstructures with fine-tuned nonlinear material properties
• Astronomia ex machina: a history, primer, and outlook on neural networks in astronomy
• Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
• Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction
• Lossy Image Compression with Quantized Hierarchical VAEs
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• Recommender systems based on graph embedding techniques: A comprehensive review
• Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior
• FitVid: Overfitting in Pixel-Level Video Prediction
• Amortized Variational Inference: A Systematic Review
• Machine Learning with Confidential Computing: A Systematization of Knowledge
• NeRF-VAE: A Geometry Aware 3D Scene Generative Model
• Perfect density models cannot guarantee anomaly detection
• Deep Generative Domain Adaptation with Temporal Attention for Cross-User Activity Recognition
• Self-Supervised Variational Auto-Encoders
• Coupled VAE: Improved Accuracy and Robustness of a Variational Autoencoder
• A multimodal dynamical variational autoencoder for audiovisual speech representation learning
• Non Gaussian Denoising Diffusion Models
• RG-Flow: A hierarchical and explainable flow model based on renormalization group and sparse prior
• Unsupervisedly Prompting AlphaFold2 for Few-Shot Learning of Accurate Folding Landscape and Protein Structure Prediction
• Score-based Generative Modeling in Latent Space
• Clockwork Variational Autoencoders
• Learning and controlling the source-filter representation of speech with a variational autoencoder
• Generative Neurosymbolic Machines
• EC-GAN: Low-Sample Classification using Semi-Supervised Algorithms and GANs
• Diffusion Priors In Variational Autoencoders
• Innovations Autoencoder and its Application in One-class Anomalous Sequence Detection
• Generalization and Memorization: The Bias Potential Model
• Spatial Dependency Networks: Neural Layers for Improved Generative Image Modeling
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Inverse problem regularization with hierarchical variational autoencoders
• NVAE-GAN Based Approach for Unsupervised Time Series Anomaly Detection
• Probabilistic Programming with Programmable Variational Inference
• Variational Autoencoders for Jet Simulation
• Denoising Diffusion Gamma Models
• On Attribution of Deepfakes
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• REX: Revisiting Budgeted Training with an Improved Schedule
• A survey on Variational Autoencoders from a GreenAI perspective
• Re-parameterizing VAEs for stability
• SCHA-VAE: Hierarchical Context Aggregation for Few-Shot Generation
• HireVAE: An Online and Adaptive Factor Model Based on Hierarchical and Regime-Switch VAE
• Face Reconstruction with Variational Autoencoder and Face Masks
• Contrastive Unpaired Translation using Focal Loss for Patch Classification
• High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve
• Hierarchical Autoencoder-based Lossy Compression for Large-scale High-resolution Scientific Data
• Unpaired Image-to-Image Translation via Latent Energy Transport
• Scarce Data Driven Deep Learning of Drones via Generalized Data Distribution Space
• Diverse Plausible Shape Completions from Ambiguous Depth Images
• Iterative VAE as a predictive brain model for out-of-distribution generalization
• Generating Unobserved Alternatives
• A Binded VAE for Inorganic Material Generation
• MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining
• Data-Driven Stochastic Motion Evaluation and Optimization with Image by Spatially-Aligned Temporal Encoding
• Deep learning and differential equations for modeling changes in individual-level latent dynamics between observation periods
• Iterative Window Mean Filter: Thwarting Diffusion-based Adversarial Purification
• Laplacian Pyramid-like Autoencoder
• Diagnosing Vulnerability of Variational Auto-Encoders to Adversarial Attacks
• Symmetric Wasserstein Autoencoders
• An Introduction to Variational Inference
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Towards disease-aware image editing of chest X-rays
• A Quaternion-Valued Variational Autoencoder
• Unsupervised Learning of Neurosymbolic Encoders
• Lossless Compression with Probabilistic Circuits
• Variational Predictive Routing with Nested Subjective Timescales
• Boxhead: A Dataset for Learning Hierarchical Representations
• FDR Control for Online Anomaly Detection
• Stay Positive: Non-Negative Image Synthesis for Augmented Reality
• Synthesizing Photorealistic Images with Deep Generative Learning
• Generation and Simulation of Yeast Microscopy Imagery with Deep Learning
• Cascading Modular Network (CAM-Net) for Multimodal Image Synthesis
• NP-DRAW: A Non-Parametric Structured Latent Variable Model for Image Generation
• Lossless Compression with Latent Variable Models
• Zero-Shot Translation using Diffusion Models
• A moment-matching metric for latent variable generative models