Ladder Variational Autoencoders
arXiv:1602.02282
Abstract
Variational Autoencoders are powerful models for unsupervised learning. However deep models with several layers of dependent stochastic variables are difficult to train which limits the improvements obtained using these highly expressive models. We propose a new inference model, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. We show that this model provides state of the art predictive log-likelihood and tighter log-likelihood lower bound compared to the purely bottom-up inference in layered Variational Autoencoders and other generative models. We provide a detailed analysis of the learned hierarchical latent representation and show that our new inference model is qualitatively different and utilizes a deeper more distributed hierarchy of latent variables. Finally, we observe that batch normalization and deterministic warm-up (gradually turning on the KL-term) are crucial for training variational models with many stochastic layers.
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Pixel Recurrent Neural Networks
- Semi-Supervised Learning with Deep Generative Models
- Theano: new features and speed improvements
- DRAW: A Recurrent Neural Network For Image Generation
- Semi-Supervised Learning with Ladder Networks
- Auxiliary Deep Generative Models
- Iterative Neural Autoregressive Distribution Estimator (NADE-k)
- The Variational Gaussian Process
Cited by in corpus (36)
- An Introduction to Variational Autoencoders
- Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
- Bayesian Compression for Deep Learning
- A Review on Deep Learning Techniques for Video Prediction
- Auxiliary Deep Generative Models
- Improving Variational Inference with Inverse Autoregressive Flow
- Towards a Neural Statistician
- Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation
- Compressing Neural Networks using the Variational Information Bottleneck
- Deep Knockoffs
- Scalable Population Synthesis with Deep Generative Modeling
- Improving Variational Auto-Encoders using Householder Flow
- Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data
- Deblending galaxies with Variational Autoencoders: a joint multi-band, multi-instrument approach
- Tackling Over-pruning in Variational Autoencoders
- Discrete Variational Autoencoders
- Fixing a Broken ELBO
- Avoiding Latent Variable Collapse With Generative Skip Models
- Universal audio synthesizer control with normalizing flows
- The Usual Suspects? Reassessing Blame for VAE Posterior Collapse
- Fast amortized inference of neural activity from calcium imaging data with variational autoencoders
- Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics
- Evaluating deep variational autoencoders trained on pan-cancer gene expression
- Modulated Variational auto-Encoders for many-to-many musical timbre transfer
- MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
- Hidden Talents of the Variational Autoencoder
- SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representations
- A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features
- mu-Forcing: Training Variational Recurrent Autoencoders for Text Generation
- A Variational Time Series Feature Extractor for Action Prediction
- Improving Bi-directional Generation between Different Modalities with Variational Autoencoders
- CVAD: A generic medical anomaly detector based on Cascade VAE
- Evaluating for Diversity in Question Generation over Text
- Deep Dive into Semi-Supervised ELBO for Improving Classification Performance
- Boxhead: A Dataset for Learning Hierarchical Representations
- From abstract items to latent spaces to observed data and back: Compositional Variational Auto-Encoder