Understanding disentangling in $\beta$-VAE

Abstract

We present new intuitions and theoretical assessments of the emergence of disentangled representation in variational autoencoders. Taking a rate-distortion theory perspective, we show the circumstances under which representations aligned with the underlying generative factors of variation of data emerge when optimising the modified ELBO bound in $\beta$-VAE, as training progresses. From these insights, we propose a modification to the training regime of $\beta$-VAE, that progressively increases the information capacity of the latent code during training. This modification facilitates the robust learning of disentangled representations in $\beta$-VAE, without the previous trade-off in reconstruction accuracy.

arXiv commentPresented at the 2017 NIPS Workshop on Learning Disentangled Representations

Cited by in corpus (111)

• Disentangled Representation Learning in Cardiac Image Analysis
• MONet: Unsupervised Scene Decomposition and Representation
• Causal Confusion in Imitation Learning
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Learning Disentangled Representations in the Imaging Domain
• Information Constraints on Auto-Encoding Variational Bayes
• Disentangling Hate in Online Memes
• Learning Robust Representations via Multi-View Information Bottleneck
• Recurrent Independent Mechanisms
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Lifelong Teacher-Student Network Learning
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• Where is the Information in a Deep Neural Network?
• Learning to Simulate High Energy Particle Collisions from Unlabeled Data
• Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis
• Generating Multivariate Load States Using a Conditional Variational Autoencoder
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• Networked Time Series Imputation via Position-aware Graph Enhanced Variational Autoencoders
• Diffusion Variational Autoencoders
• Amortized Variational Inference: A Systematic Review
• Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness
• A Spectral Regularizer for Unsupervised Disentanglement
• The Next Big Thing(s) in Unsupervised Machine Learning: Five Lessons from Infant Learning
• Normalizing Flows for High-Dimensional Detector Simulations
• Multi-Scale and Multi-Layer Contrastive Learning for Domain Generalization
• Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems
• Learning-Aided Physical Layer Attacks Against Multicarrier Communications in IoT
• An Identifiable Double VAE For Disentangled Representations
• Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images
• PAC-Bayes Information Bottleneck
• Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
• Learning Group Structure and Disentangled Representations of Dynamical Environments
• VAE-Info-cGAN: Generating Synthetic Images by Combining Pixel-level and Feature-level Geospatial Conditional Inputs
• Balancing Reconstruction Quality and Regularisation in ELBO for VAEs
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• Disentangled Representation Learning with Wasserstein Total Correlation
• Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations
• Disentangled Representations from Non-Disentangled Models
• Disentangling and Learning Robust Representations with Natural Clustering
• Learning Cross-domain Generalizable Features by Representation Disentanglement
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Deep learning insights into non-universality in the halo mass function
• Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case
• Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
• Multi-Task Variational Information Bottleneck
• SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations
• Learning latent representations for style control and transfer in end-to-end speech synthesis
• Towards a Hypothesis on Visual Transformation based Self-Supervision
• Information Theoretic Meta Learning with Gaussian Processes
• A Fine-Grained Analysis on Distribution Shift
• dMelodies: A Music Dataset for Disentanglement Learning
• On Disentanglement in Gaussian Process Variational Autoencoders
• Testing Outlier Detection Algorithms for Identifying Early-Stage Solute Clusters in Atom Probe Tomography
• Exploring Story Generation with Multi-task Objectives in Variational Autoencoders
• Variational Feature Disentangling for Fine-Grained Few-Shot Classification
• Generating 3D People in Scenes without People
• Walsh-Hadamard Variational Inference for Bayesian Deep Learning
• Training Interpretable Convolutional Neural Networks by Differentiating Class-specific Filters
• How to Not Measure Disentanglement
• Achieving Explainability for Plant Disease Classification with Disentangled Variational Autoencoders
• PRI-VAE: Principle-of-Relevant-Information Variational Autoencoders
• On the Robustness to Misspecification of $\alpha$-Posteriors and Their Variational Approximations
• Macro Action Reinforcement Learning with Sequence Disentanglement using Variational Autoencoder
• Discond-VAE: Disentangling Continuous Factors from the Discrete
• Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
• Representation Disentanglement for Multi-task Learning with application to Fetal Ultrasound
• On the Transferability of VAE Embeddings using Relational Knowledge with Semi-Supervision
• Towards a Theoretical Understanding of the Robustness of Variational Autoencoders
• Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval
• Deep Anomaly Detection by Residual Adaptation
• WeLa-VAE: Learning Alternative Disentangled Representations Using Weak Labels
• Disentangled Motif-aware Graph Learning for Phrase Grounding
• Learning to Manipulate Individual Objects in an Image
• Variation Network: Learning High-level Attributes for Controlled Input Manipulation
• Deciphering galaxy images using machine vision -- Combining variational autoencoder and principal component analysis for feature extraction
• $\beta$-Annealed Variational Autoencoder for glitches
• From Rain Generation to Rain Removal
• Mixture Representation Learning with Coupled Autoencoders
• Unsupervised heart abnormality detection based on phonocardiogram analysis with Beta Variational Auto-Encoders
• Bridging the ELBO and MMD
• Semi-Supervised Disentanglement of Class-Related and Class-Independent Factors in VAE
• Cauchy-Schwarz Regularized Autoencoder
• To Regularize or Not To Regularize? The Bias Variance Trade-off in Regularized AEs
• Learning and Inference in Imaginary Noise Models
• Disentangled Neural Architecture Search
• Modeling the Probabilistic Distribution of Unlabeled Data forOne-shot Medical Image Segmentation
• Complex Skill Acquisition Through Simple Skill Imitation Learning
• Disentangling Latent Space for VAE by Label Relevant/Irrelevant Dimensions
• Improve variational autoEncoder with auxiliary softmax multiclassifier
• Mutual Information-based Disentangled Neural Networks for Classifying Unseen Categories in Different Domains: Application to Fetal Ultrasound Imaging
• Robust Disentanglement of a Few Factors at a Time
• The Neglected Sibling: Isotropic Gaussian Posterior for VAE
• Quantifying and Learning Linear Symmetry-Based Disentanglement
• Temporal Clustering with External Memory Network for Disease Progression Modeling
• A Deeper Look at the Unsupervised Learning of Disentangled Representations in $\beta$-VAE from the Perspective of Core Object Recognition
• Learning to Learn Graph Topologies
• Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders
• Are there any 'object detectors' in the hidden layers of CNNs trained to identify objects or scenes?
• Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders
• Kullback-Leibler Divergence-Based Out-of-Distribution Detection with Flow-Based Generative Models
• Lifelong Mixture of Variational Autoencoders
• SepVAE: a contrastive VAE to separate pathological patterns from healthy ones
• Towards Better Understanding of Disentangled Representations via Mutual Information
• Joint Estimation of Image Representations and their Lie Invariants
• Variational Information Bottleneck Model for Accurate Indoor Position Recognition
• Variational Auto-Encoder Architectures that Excel at Causal Inference