MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders

Abstract

Variational Autoencoder (VAE), a simple and effective deep generative model, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations. However, recent studies demonstrate that, when equipped with expressive generative distributions (aka. decoders), VAE suffers from learning uninformative latent representations with the observation called KL Varnishing, in which case VAE collapses into an unconditional generative model. In this work, we introduce mutual posterior-divergence regularization, a novel regularization that is able to control the geometry of the latent space to accomplish meaningful representation learning, while achieving comparable or superior capability of density estimation. Experiments on three image benchmark datasets demonstrate that, when equipped with powerful decoders, our model performs well both on density estimation and representation learning.

Published at ICLR-2019. 12 pages contents + 4 pages appendix, 5 figures

References in corpus (13)

• Auto-Encoding Variational Bayes
• Pixel Recurrent Neural Networks
• Semi-Supervised Learning with Deep Generative Models
• Density estimation using Real NVP
• Generative Moment Matching Networks
• InfoVAE: Information Maximizing Variational Autoencoders
• MADE: Masked Autoencoder for Distribution Estimation
• Variational Lossy Autoencoder
• Ladder Variational Autoencoders
• PixelSNAIL: An Improved Autoregressive Generative Model
• Discrete Variational Autoencoders
• Semi-Amortized Variational Autoencoders
• Latent Variable Modeling with Diversity-Inducing Mutual Angular Regularization

Cited by in corpus (9)

• NVAE: A Deep Hierarchical Variational Autoencoder
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Score-based Generative Modeling in Latent Space
• MaCow: Masked Convolutional Generative Flow
• A Batch Normalized Inference Network Keeps the KL Vanishing Away
• Decoupling Global and Local Representations via Invertible Generative Flows
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory
• Improve variational autoEncoder with auxiliary softmax multiclassifier
• Noise Contrastive Variational Autoencoders