Importance Weighted Autoencoders

Abstract

The variational autoencoder (VAE; Kingma, Welling (2014)) is a recently proposed generative model pairing a top-down generative network with a bottom-up recognition network which approximates posterior inference. It typically makes strong assumptions about posterior inference, for instance that the posterior distribution is approximately factorial, and that its parameters can be approximated with nonlinear regression from the observations. As we show empirically, the VAE objective can lead to overly simplified representations which fail to use the network's entire modeling capacity. We present the importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting. In the IWAE, the recognition network uses multiple samples to approximate the posterior, giving it increased flexibility to model complex posteriors which do not fit the VAE modeling assumptions. We show empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log-likelihood on density estimation benchmarks.

Submitted to ICLR 2015

Cited by in corpus (54)

• An Introduction to Variational Autoencoders
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Tree Tensor Networks for Generative Modeling
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework
• Deep Knockoffs
• Lifelong Generative Modeling
• Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder
• Diagnosing and Enhancing VAE Models
• Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data
• Deep Learning of Nonnegativity-Constrained Autoencoders for Enhanced Understanding of Data
• Anomaly Detection for Skin Disease Images Using Variational Autoencoder
• Metrics for Deep Generative Models
• Spatiotemporal Tensor Completion for Improved Urban Traffic Imputation
• Physics-aware Reduced-order Modeling of Transonic Flow via $\beta$-Variational Autoencoder
• Fixing a Broken ELBO
• Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
• Semi-Amortized Variational Autoencoders
• Creativity: Generating Diverse Questions using Variational Autoencoders
• A Tutorial on Deep Latent Variable Models of Natural Language
• On the challenges of learning with inference networks on sparse, high-dimensional data
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Amortized Variational Inference: A Systematic Review
• A Contrastive Variational Graph Auto-Encoder for Node Clustering
• Advances in Variational Inference
• Self-Supervised Variational Auto-Encoders
• not-MIWAE: Deep Generative Modelling with Missing not at Random Data
• Learning Wake-Sleep Recurrent Attention Models
• Generative Latent Flow
• Deep Gaussian Process-Based Bayesian Inference for Contaminant Source Localization
• Energy-based Models for Video Anomaly Detection
• Deep Generative Imputation Model for Missing Not At Random Data
• Degeneration in VAE: in the Light of Fisher Information Loss
• Building Surrogate Models of Nuclear Density Functional Theory with Gaussian Processesand Autoencoders
• Joint Distributions for TensorFlow Probability
• Balancing Reconstruction Quality and Regularisation in ELBO for VAEs
• Learning Deep Generative Models with Doubly Stochastic MCMC
• Disentangled Representations in Neural Models
• Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting
• Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case
• Continuous Mixtures of Tractable Probabilistic Models
• Deep Variational Inference Without Pixel-Wise Reconstruction
• Generative Machine Learning for Multivariate Equity Returns
• Do sequence-to-sequence VAEs learn global features of sentences?
• Learning Flat Latent Manifolds with VAEs
• Variational Open-Domain Question Answering
• Model-agnostic out-of-distribution detection using combined statistical tests
• Max-Margin Deep Generative Models for (Semi-)Supervised Learning
• Parameterization of Forced Isotropic Turbulent Flow using Autoencoders and Generative Adversarial Networks
• Density Deconvolution with Normalizing Flows
• AE-OT-GAN: Training GANs from data specific latent distribution
• Multi-Source Neural Variational Inference
• Stochastic Approximate Gradient Descent via the Langevin Algorithm
• Explainability as statistical inference
• Embedding-reparameterization procedure for manifold-valued latent variables in generative models