Improving Variational Inference with Inverse Autoregressive Flow

Abstract

The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network. In experiments, we show that IAF significantly improves upon diagonal Gaussian approximate posteriors. In addition, we demonstrate that a novel type of variational autoencoder, coupled with IAF, is competitive with neural autoregressive models in terms of attained log-likelihood on natural images, while allowing significantly faster synthesis.

References in corpus (9)

• WaveNet: A Generative Model for Raw Audio
• Conditional Image Generation with PixelCNN Decoders
• Wide Residual Networks
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• MADE: Masked Autoencoder for Distribution Estimation
• Variational Bayesian Inference with Stochastic Search
• Towards Conceptual Compression

Cited by in corpus (94)

• Conditional Image Synthesis With Auxiliary Classifier GANs
• Density estimation using Real NVP
• Adversarially Learned Inference
• Unsupervised Image-to-Image Translation Networks
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• Deep Learning and its Application to LHC Physics
• InfoVAE: Information Maximizing Variational Autoencoders
• Causal Effect Inference with Deep Latent-Variable Models
• Towards Deeper Understanding of Variational Autoencoding Models
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework
• Tag N' Train: A Technique to Train Improved Classifiers on Unlabeled Data
• Deep Active Inference
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• Inference Suboptimality in Variational Autoencoders
• A-NICE-MC: Adversarial Training for MCMC
• Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data
• Variational Inference using Implicit Distributions
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• Dual Adversarial Auto-Encoders for Clustering
• WaveFlow: A Compact Flow-based Model for Raw Audio
• Learning Likelihoods with Conditional Normalizing Flows
• Challenges for Unsupervised Anomaly Detection in Particle Physics
• Improving Variational Autoencoders for New Physics Detection at the LHC with Normalizing Flows
• Tackling Over-pruning in Variational Autoencoders
• Predictive Collective Variable Discovery with Deep Bayesian Models
• Dynamical mass inference of galaxy clusters with neural flows
• Fixing a Broken ELBO
• Adversarially Regularized Autoencoders
• Copula Flows for Synthetic Data Generation
• Differentiable Strong Lensing: Uniting Gravity and Neural Nets through Differentiable Probabilistic Programming
• Learning Hierarchical Features from Generative Models
• A Tutorial on Deep Latent Variable Models of Natural Language
• Simulation-based inference of dynamical galaxy cluster masses with 3D convolutional neural networks
• CaloFlow II: Even Faster and Still Accurate Generation of Calorimeter Showers with Normalizing Flows
• Learning to Draw Samples with Amortized Stein Variational Gradient Descent
• Tighter Variational Bounds are Not Necessarily Better
• Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models
• Dr.VAE: Drug Response Variational Autoencoder
• Variational Memory Addressing in Generative Models
• Learnable Explicit Density for Continuous Latent Space and Variational Inference
• Transformation Autoregressive Networks
• Tensorial Mixture Models
• Machine Learning Statistical Gravity from Multi-Region Entanglement Entropy
• A Generative Model of Galactic Dust Emission Using Variational Inference
• Degeneration in VAE: in the Light of Fisher Information Loss
• Explicit Disentanglement of Appearance and Perspective in Generative Models
• Preventing Posterior Collapse with delta-VAEs
• DeLiGAN : Generative Adversarial Networks for Diverse and Limited Data
• Improving Sampling from Generative Autoencoders with Markov Chains
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• Densely connected normalizing flows
• Sparsity in Variational Autoencoders
• Uncertainty in Multitask Transfer Learning
• Balancing Reconstruction Quality and Regularisation in ELBO for VAEs
• Diffusion Priors In Variational Autoencoders
• Innovations Autoencoder and its Application in One-class Anomalous Sequence Detection
• Unsupervised Multi-Domain Image Translation with Domain-Specific Encoders/Decoders
• Compound Probabilistic Context-Free Grammars for Grammar Induction
• Trumpets: Injective Flows for Inference and Inverse Problems
• Recursive Inference for Variational Autoencoders
• tvGP-VAE: Tensor-variate Gaussian Process Prior Variational Autoencoder
• Improving VAEs' Robustness to Adversarial Attack
• Failure Modes of Variational Autoencoders and Their Effects on Downstream Tasks
• Hybrid VAE: Improving Deep Generative Models using Partial Observations
• Hierarchical Sequence to Sequence Voice Conversion with Limited Data
• mu-Forcing: Training Variational Recurrent Autoencoders for Text Generation
• Learning Flat Latent Manifolds with VAEs
• Sparse Flows: Pruning Continuous-depth Models
• Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models
• Semi-unsupervised Learning of Human Activity using Deep Generative Models
• Reducing the Amortization Gap in Variational Autoencoders: A Bayesian Random Function Approach
• Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
• A novel stellar spectrum denoising method based on deep Bayesian modeling
• Featurized Density Ratio Estimation
• MeshODE: A Robust and Scalable Framework for Mesh Deformation
• MIM: Mutual Information Machine
• NWT: Towards natural audio-to-video generation with representation learning
• LaDDer: Latent Data Distribution Modelling with a Generative Prior
• Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows
• ByPE-VAE: Bayesian Pseudocoresets Exemplar VAE
• TzK Flow - Conditional Generative Model
• Mixture Representation Learning with Coupled Autoencoders
• On Investigation of Unsupervised Speech Factorization Based on Normalization Flow
• Causal Autoregressive Flows
• Deep Learning on Attributed Graphs: A Journey from Graphs to Their Embeddings and Back
• A Contemporary Overview of Probabilistic Latent Variable Models
• Noise Contrastive Variational Autoencoders
• Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning
• High Mutual Information in Representation Learning with Symmetric Variational Inference
• TzK: Flow-Based Conditional Generative Model
• Improving the expressiveness of neural vocoding with non-affine Normalizing Flows
• Hierarchical Reinforcement Learning with Optimal Level Synchronization Based on Flow-Based Deep Generative Model
• Enhancing audio quality for expressive Neural Text-to-Speech
• Molecular Attributes Transfer from Non-Parallel Data