Disentangling Disentanglement in Variational Autoencoders

Abstract

We develop a generalisation of disentanglement in VAEs---decomposition of the latent representation---characterising it as the fulfilment of two factors: a) the latent encodings of the data having an appropriate level of overlap, and b) the aggregate encoding of the data conforming to a desired structure, represented through the prior. Decomposition permits disentanglement, i.e. explicit independence between latents, as a special case, but also allows for a much richer class of properties to be imposed on the learnt representation, such as sparsity, clustering, independent subspaces, or even intricate hierarchical dependency relationships. We show that the $\beta$-VAE varies from the standard VAE predominantly in its control of latent overlap and that for the standard choice of an isotropic Gaussian prior, its objective is invariant to rotations of the latent representation. Viewed from the decomposition perspective, breaking this invariance with simple manipulations of the prior can yield better disentanglement with little or no detriment to reconstructions. We further demonstrate how other choices of prior can assist in producing different decompositions and introduce an alternative training objective that allows the control of both decomposition factors in a principled manner.

arXiv commentAccepted for publication at ICML 2019

References in corpus (17)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Learning deep representations by mutual information estimation and maximization
• On the Convergence of Adam and Beyond
• Semi-Supervised Learning with Deep Generative Models
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• InfoVAE: Information Maximizing Variational Autoencoders
• Towards a Definition of Disentangled Representations
• Variational Lossy Autoencoder
• Disentangling factors of variation in deep representations using adversarial training
• Composing graphical models with neural networks for structured representations and fast inference
• Isolating Sources of Disentanglement in Variational Autoencoders
• Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations
• Discovering Hidden Factors of Variation in Deep Networks
• Importance Weighting and Variational Inference
• On Nesting Monte Carlo Estimators

Cited by in corpus (12)

• Variational Autoencoders and Nonlinear ICA: A Unifying Framework
• Disentangling Factors of Variation Using Few Labels
• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition
• Joint embedding of structure and features via graph convolutional networks
• Deep active inference agents using Monte-Carlo methods
• On the Fairness of Disentangled Representations
• Sparse Representation Learning with Modified q-VAE towards Minimal Realization of World Model
• Jigsaw-VAE: Towards Balancing Features in Variational Autoencoders
• Multi-type Disentanglement without Adversarial Training
• The Variational InfoMax Learning Objective
• Robust Face Verification via Disentangled Representations