Tackling Over-pruning in Variational Autoencoders

Abstract

Variational autoencoders (VAE) are directed generative models that learn factorial latent variables. As noted by Burda et al. (2015), these models exhibit the problem of factor over-pruning where a significant number of stochastic factors fail to learn anything and become inactive. This can limit their modeling power and their ability to learn diverse and meaningful latent representations. In this paper, we evaluate several methods to address this problem and propose a more effective model-based approach called the epitomic variational autoencoder (eVAE). The so-called epitomes of this model are groups of mutually exclusive latent factors that compete to explain the data. This approach helps prevent inactive units since each group is pressured to explain the data. We compare the approaches with qualitative and quantitative results on MNIST and TFD datasets. Our results show that eVAE makes efficient use of model capacity and generalizes better than VAE.

References in corpus (3)

• Deep Convolutional Inverse Graphics Network
• Better Mixing via Deep Representations
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Cited by in corpus (7)

• The Usual Suspects? Reassessing Blame for VAE Posterior Collapse
• Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
• Assisted Sound Sample Generation with Musical Conditioning in Adversarial Auto-Encoders
• Jigsaw-VAE: Towards Balancing Features in Variational Autoencoders
• REX: Revisiting Budgeted Training with an Improved Schedule
• Improved Training of Sparse Coding Variational Autoencoder via Weight Normalization
• Genetic Constrained Graph Variational Autoencoder for COVID-19 Drug Discovery