Auto-Encoding Total Correlation Explanation

Abstract

Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned. We propose an information-theoretic approach to characterizing disentanglement and dependence in representation learning using multivariate mutual information, also called total correlation. The principle of total Cor-relation Ex-planation (CorEx) has motivated successful unsupervised learning applications across a variety of domains, but under some restrictive assumptions. Here we relax those restrictions by introducing a flexible variational lower bound to CorEx. Surprisingly, we find that this lower bound is equivalent to the one in variational autoencoders (VAE) under certain conditions. This information-theoretic view of VAE deepens our understanding of hierarchical VAE and motivates a new algorithm, AnchorVAE, that makes latent codes more interpretable through information maximization and enables generation of richer and more realistic samples.

References in corpus (11)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Opening the Black Box of Deep Neural Networks via Information
• Disentangling by Factorising
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• InfoVAE: Information Maximizing Variational Autoencoders
• Independently Controllable Factors
• Independently Controllable Features
• Learning Hierarchical Features from Generative Models

Cited by in corpus (13)

• Isolating Sources of Disentanglement in Variational Autoencoders
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Lifelong Teacher-Student Network Learning
• Learning Disentangled Joint Continuous and Discrete Representations
• Unsupervised Belief Representation Learning with Information-Theoretic Variational Graph Auto-Encoders
• A Tutorial on Deep Latent Variable Models of Natural Language
• Dual Swap Disentangling
• InfoCatVAE: Representation Learning with Categorical Variational Autoencoders
• Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling
• Disentangling and Learning Robust Representations with Natural Clustering
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• Invariant Representations with Stochastically Quantized Neural Networks
• Multi-view Alignment and Generation in CCA via Consistent Latent Encoding