Towards Better Understanding of Disentangled Representations via Mutual Information

Abstract

Most existing works on disentangled representation learning are solely built upon an marginal independence assumption: all factors in disentangled representations should be statistically independent. This assumption is necessary but definitely not sufficient for the disentangled representations without additional inductive biases in the modeling process, which is shown theoretically in recent studies. We argue in this work that disentangled representations should be characterized by their relation with observable data. In particular, we formulate such a relation through the concept of mutual information: the mutual information between each factor of the disentangled representations and data should be invariant conditioned on values of the other factors. Together with the widely accepted independence assumption, we further bridge it with the conditional independence of factors in representations conditioned on data. Moreover, we note that conditional independence of latent variables has been imposed on most VAE-type models and InfoGAN due to the artificial choice of factorized approximate posterior $q(\rvz|\rvx)$ in the encoders. Such an arrangement of encoders introduces a crucial inductive bias for disentangled representations. To demonstrate the importance of our proposed assumption and the related inductive bias, we show in experiments that violating the assumption leads to decline of disentanglement among factors in the learned representations.

References in corpus (20)

• Auto-Encoding Variational Bayes
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• Recent Advances in Autoencoder-Based Representation Learning
• Deep Variational Information Bottleneck
• Visual Reinforcement Learning with Imagined Goals
• Towards a Definition of Disentangled Representations
• Understanding disentangling in $\beta$-VAE
• Disentangling factors of variation in deep representations using adversarial training
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Learning Deep Disentangled Embeddings with the F-Statistic Loss
• A Unified Feature Disentangler for Multi-Domain Image Translation and Manipulation
• Disentangled Sequential Autoencoder
• Disentangling Factors of Variation via Generative Entangling
• Weakly Supervised Disentanglement with Guarantees
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Learning Independent Features with Adversarial Nets for Non-linear ICA
• Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations
• Theory and Evaluation Metrics for Learning Disentangled Representations
• Competitive Training of Mixtures of Independent Deep Generative Models