Improving Variational Auto-Encoders using Householder Flow

Abstract

Variational auto-encoders (VAE) are scalable and powerful generative models. However, the choice of the variational posterior determines tractability and flexibility of the VAE. Commonly, latent variables are modeled using the normal distribution with a diagonal covariance matrix. This results in computational efficiency but typically it is not flexible enough to match the true posterior distribution. One fashion of enriching the variational posterior distribution is application of normalizing flows, i.e., a series of invertible transformations to latent variables with a simple posterior. In this paper, we follow this line of thinking and propose a volume-preserving flow that uses a series of Householder transformations. We show empirically on MNIST dataset and histopathology data that the proposed flow allows to obtain more flexible variational posterior and competitive results comparing to other normalizing flows.

A corrected version of the paper submitted to Bayesian Deep Learning Workshop (NIPS 2016)

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• NICE: Non-linear Independent Components Estimation
• Language Modeling with Gated Convolutional Networks
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Ladder Variational Autoencoders
• Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections
• Predicting distributions with Linearizing Belief Networks

Cited by in corpus (60)

• An Introduction to Variational Autoencoders
• Normalizing Flows: An Introduction and Review of Current Methods
• Neural Ordinary Differential Equations
• Normalizing Flows for Probabilistic Modeling and Inference
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Sylvester Normalizing Flows for Variational Inference
• Deep Active Inference
• Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities
• Conditional Flow Variational Autoencoders for Structured Sequence Prediction
• Dual Adversarial Auto-Encoders for Clustering
• Topic-Guided Variational Autoencoders for Text Generation
• Trivializations for Gradient-Based Optimization on Manifolds
• A Tutorial on Deep Latent Variable Models of Natural Language
• Neural Spline Flows
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Learning Symmetries of Classical Integrable Systems
• Understanding and Mitigating Exploding Inverses in Invertible Neural Networks
• Stochastic Normalizing Flows
• Bayesian Semisupervised Learning with Deep Generative Models
• iUNets: Fully invertible U-Nets with Learnable Up- and Downsampling
• Categorical Normalizing Flows via Continuous Transformations
• Invertible DenseNets with Concatenated LipSwish
• Deep Diffeomorphic Normalizing Flows
• Deep Learning Models for Digital Pathology
• Balancing Reconstruction Quality and Regularisation in ELBO for VAEs
• Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling
• Towards Interpretable Polyphonic Transcription with Invertible Neural Networks
• Variational Autoencoder with Implicit Optimal Priors
• Recursive Inference for Variational Autoencoders
• Multimodal Generative Models for Compositional Representation Learning
• Kernel Implicit Variational Inference
• Max-Affine Spline Insights into Deep Generative Networks
• What if Neural Networks had SVDs?
• Learning normalizing flows from Entropy-Kantorovich potentials
• Unbiased Gradient Estimation for Variational Auto-Encoders using Coupled Markov Chains
• Relative gradient optimization of the Jacobian term in unsupervised deep learning
• Reducing the Amortization Gap in Variational Autoencoders: A Bayesian Random Function Approach
• The Convolution Exponential and Generalized Sylvester Flows
• Sum-Product-Transform Networks: Exploiting Symmetries using Invertible Transformations
• Backpropagation for Implicit Spectral Densities
• Modeling Item Response Theory with Stochastic Variational Inference
• Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
• MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining
• Copula-like Variational Inference
• General Invertible Transformations for Flow-based Generative Modeling
• Ordering Dimensions with Nested Dropout Normalizing Flows
• LaDDer: Latent Data Distribution Modelling with a Generative Prior
• One Reflection Suffice
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• ByPE-VAE: Bayesian Pseudocoresets Exemplar VAE
• Training Invertible Linear Layers through Rank-One Perturbations
• Differentiable Antithetic Sampling for Variance Reduction in Stochastic Variational Inference
• Cauchy-Schwarz Regularized Autoencoder
• Information Theoretic Lower Bounds on Negative Log Likelihood
• Gradient Boosted Normalizing Flows
• Approximation Based Variance Reduction for Reparameterization Gradients
• Learning and Generalization in Overparameterized Normalizing Flows
• Viscos Flows: Variational Schur Conditional Sampling With Normalizing Flows
• Enhanced Variational Inference with Dyadic Transformation
• Out-of-Distribution Detection of Melanoma using Normalizing Flows