Variational Lossy Autoencoder

Abstract

Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution $p(z)$ and decoding distribution $p(x|z)$, we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks.

Added CIFAR10 experiments; ICLR 2017

Cited by in corpus (118)

• An Introduction to Variational Autoencoders
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Mixed supervision for surface-defect detection: from weakly to fully supervised learning
• Recent Advances in Autoencoder-Based Representation Learning
• Joint Autoregressive and Hierarchical Priors for Learned Image Compression
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• Deep Appearance Models for Face Rendering
• Neural Relational Inference for Interacting Systems
• Video Compression With Rate-Distortion Autoencoders
• DisenHAN: Disentangled Heterogeneous Graph Attention Network for Recommendation
• SR2CNN: Zero-Shot Learning for Signal Recognition
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Generating and designing DNA with deep generative models
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder
• From Variational to Deterministic Autoencoders
• Diagnosing and Enhancing VAE Models
• Learning Robust Representations via Multi-View Information Bottleneck
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows
• Conditional Flow Variational Autoencoders for Structured Sequence Prediction
• VAE with a VampPrior
• Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Disentangling Disentanglement in Variational Autoencoders
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• Adversarially Regularized Autoencoders
• Universal audio synthesizer control with normalizing flows
• Semi-Amortized Variational Autoencoders
• Z-Forcing: Training Stochastic Recurrent Networks
• A Tutorial on Deep Latent Variable Models of Natural Language
• Improving Variational Encoder-Decoders in Dialogue Generation
• On the challenges of learning with inference networks on sparse, high-dimensional data
• Tighter Variational Bounds are Not Necessarily Better
• Advances in Variational Inference
• Hierarchical Autoregressive Image Models with Auxiliary Decoders
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Self-Supervised Variational Auto-Encoders
• VFlow: More Expressive Generative Flows with Variational Data Augmentation
• Review of end-to-end speech synthesis technology based on deep learning
• A Hierarchical Latent Structure for Variational Conversation Modeling
• A Review of Learning with Deep Generative Models from Perspective of Graphical Modeling
• A Tutorial on VAEs: From Bayes' Rule to Lossless Compression
• Variational Autoencoders with Normalizing Flow Decoders
• Preventing Posterior Collapse with delta-VAEs
• MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
• Deep Amortized Clustering
• Maximizing Mutual Information for Tacotron
• Score-based Generative Modeling in Latent Space
• Reweighted Expectation Maximization
• Bounded Information Rate Variational Autoencoders
• Variable-rate discrete representation learning
• Safe Interactive Model-Based Learning
• DeepScaffold: a comprehensive tool for scaffold-based de novo drug discovery using deep learning
• On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder
• Invertible Generative Modeling using Linear Rational Splines
• A Contrastive Learning Approach for Training Variational Autoencoder Priors
• Diffusion Priors In Variational Autoencoders
• Spatial Dependency Networks: Neural Layers for Improved Generative Image Modeling
• Soft then Hard: Rethinking the Quantization in Neural Image Compression
• G-VAE, a Geometric Convolutional VAE for ProteinStructure Generation
• Learning Hierarchical Priors in VAEs
• Jigsaw-VAE: Towards Balancing Features in Variational Autoencoders
• Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings
• Learning Disentangled Representations for Time Series
• mu-Forcing: Training Variational Recurrent Autoencoders for Text Generation
• Do sequence-to-sequence VAEs learn global features of sentences?
• A Variational Prosody Model for Mapping the Context-Sensitive Variation of Functional Prosodic Prototypes
• Improving Disentangled Representation Learning with the Beta Bernoulli Process
• A Batch Normalized Inference Network Keeps the KL Vanishing Away
• Deterministic Decoding for Discrete Data in Variational Autoencoders
• Improved Variational Neural Machine Translation by Promoting Mutual Information
• Importance Weighted Hierarchical Variational Inference
• Representation Disentanglement for Multi-task Learning with application to Fetal Ultrasound
• HAVANA: Hierarchical and Variation-Normalized Autoencoder for Person Re-identification
• Customizing Sequence Generation with Multi-Task Dynamical Systems
• Rate-Regularization and Generalization in VAEs
• Gradient Origin Networks
• Unsupervised Multi-Domain Multimodal Image-to-Image Translation with Explicit Domain-Constrained Disentanglement
• Optimal Variance Control of the Score Function Gradient Estimator for Importance Weighted Bounds
• Likelihood Assignment for Out-of-Distribution Inputs in Deep Generative Models is Sensitive to Prior Distribution Choice
• Likelihood Contribution based Multi-scale Architecture for Generative Flows
• Generating Dialogue Responses from a Semantic Latent Space
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• MIM: Mutual Information Machine
• Density Deconvolution with Normalizing Flows
• Improving latent variable descriptiveness with AutoGen
• Holographic Neural Architectures
• Discrete Point Flow Networks for Efficient Point Cloud Generation
• Residual-Guided In-Loop Filter Using Convolution Neural Network
• Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
• A Factorial Mixture Prior for Compositional Deep Generative Models
• Decoupling Global and Local Representations via Invertible Generative Flows
• Normalizing Flows with Multi-Scale Autoregressive Priors
• Variation Network: Learning High-level Attributes for Controlled Input Manipulation
• Deep Latent-Variable Kernel Learning
• Variance Constrained Autoencoding
• Color Visual Illusions: A Statistics-based Computational Model
• Benefiting Deep Latent Variable Models via Learning the Prior and Removing Latent Regularization
• Gradient Boosted Normalizing Flows
• $\rho$-VAE: Autoregressive parametrization of the VAE encoder
• Increasing the Generalisation Capacity of Conditional VAEs
• A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
• Improve variational autoEncoder with auxiliary softmax multiclassifier
• Regularization with Latent Space Virtual Adversarial Training
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory
• Estimating Disentangled Belief about Hidden State and Hidden Task for Meta-RL
• Entropy optimized semi-supervised decomposed vector-quantized variational autoencoder model based on transfer learning for multiclass text classification and generation
• Nana-HDR: A Non-attentive Non-autoregressive Hybrid Model for TTS
• Mutual Information-based Disentangled Neural Networks for Classifying Unseen Categories in Different Domains: Application to Fetal Ultrasound Imaging
• Knowledge Generation -- Variational Bayes on Knowledge Graphs
• Neural representation and generation for RNA secondary structures
• Self-Reflective Variational Autoencoder
• Variational Composite Autoencoders
• High Mutual Information in Representation Learning with Symmetric Variational Inference