Discrete Variational Autoencoders

Abstract

Probabilistic models with discrete latent variables naturally capture datasets composed of discrete classes. However, they are difficult to train efficiently, since backpropagation through discrete variables is generally not possible. We present a novel method to train a class of probabilistic models with discrete latent variables using the variational autoencoder framework, including backpropagation through the discrete latent variables. The associated class of probabilistic models comprises an undirected discrete component and a directed hierarchical continuous component. The discrete component captures the distribution over the disconnected smooth manifolds induced by the continuous component. As a result, this class of models efficiently learns both the class of objects in an image, and their specific realization in pixels, from unsupervised data, and outperforms state-of-the-art methods on the permutation-invariant MNIST, Omniglot, and Caltech-101 Silhouettes datasets.

Published as a conference paper at ICLR 2017

References in corpus (9)

• DRAW: A Recurrent Neural Network For Image Generation
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Ladder Variational Autoencoders
• Composing graphical models with neural networks for structured representations and fast inference
• Variational inference for Monte Carlo objectives
• Variational Bayesian Inference with Stochastic Search
• Techniques for Learning Binary Stochastic Feedforward Neural Networks
• Stochastic Backpropagation through Mixture Density Distributions
• A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features

Cited by in corpus (23)

• NVAE: A Deep Hierarchical Variational Autoencoder
• Learning Sparse Neural Networks through $L_0$ Regularization
• Lossy Image Compression with Quantized Hierarchical VAEs
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• All SMILES Variational Autoencoder
• DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
• Simple and Effective VAE Training with Calibrated Decoders
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• Biased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitations
• Associative Compression Networks for Representation Learning
• PixelVAE++: Improved PixelVAE with Discrete Prior
• MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
• From Ansätze to Z-gates: a NASA View of Quantum Computing
• A Contrastive Learning Approach for Training Variational Autoencoder Priors
• Improved Gradient-Based Optimization Over Discrete Distributions
• Direct Optimization through $\arg \max$ for Discrete Variational Auto-Encoder
• Learning Latent Space Energy-Based Prior Model
• Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip
• Stochastic Sequential Neural Networks with Structured Inference
• Neural Variational Inference and Learning in Undirected Graphical Models
• Inherent Weight Normalization in Stochastic Neural Networks
• Unsupervised Learning of Neurosymbolic Encoders
• RBM-Flow and D-Flow: Invertible Flows with Discrete Energy Base Spaces