DVAE++: Discrete Variational Autoencoders with Overlapping Transformations

Abstract

Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors. We derive a new variational bound to efficiently train with Boltzmann machine priors. Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables. Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al., 2016; Jang et al., 2016), and discrete variational autoencoders (Rolfe 2016).

Published as a conference paper at International Conference on Machine Learning (ICML), 2018

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Categorical Reparameterization with Gumbel-Softmax
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Pixel Recurrent Neural Networks
• Semi-Supervised Learning with Deep Generative Models
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Hierarchical Multiscale Recurrent Neural Networks
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Variational inference for Monte Carlo objectives
• VAE with a VampPrior
• Discrete Variational Autoencoders
• MuProp: Unbiased Backpropagation for Stochastic Neural Networks

Cited by in corpus (21)

• NVAE: A Deep Hierarchical Variational Autoencoder
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Learning Neural-Symbolic Descriptive Planning Models via Cube-Space Priors: The Voyage Home (to STRIPS)
• PixelVAE++: Improved PixelVAE with Discrete Prior
• Score-based Generative Modeling in Latent Space
• Estimation of Dimensions Contributing to Detected Anomalies with Variational Autoencoders
• A Contrastive Learning Approach for Training Variational Autoencoder Priors
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• Direct Optimization through $\arg \max$ for Discrete Variational Auto-Encoder
• Improved Gradient-Based Optimization Over Discrete Distributions
• Learning Latent Space Energy-Based Prior Model
• Neural-Symbolic Descriptive Action Model from Images: The Search for STRIPS
• Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders
• Probabilistic Circuits for Variational Inference in Discrete Graphical Models
• The ELBO of Variational Autoencoders Converges to a Sum of Three Entropies
• Sparse Communication via Mixed Distributions
• Amortised Learning by Wake-Sleep
• Reliable Categorical Variational Inference with Mixture of Discrete Normalizing Flows
• RBM-Flow and D-Flow: Invertible Flows with Discrete Energy Base Spaces