Disentangling by Factorising

Abstract

We define and address the problem of unsupervised learning of disentangled representations on data generated from independent factors of variation. We propose FactorVAE, a method that disentangles by encouraging the distribution of representations to be factorial and hence independent across the dimensions. We show that it improves upon $\beta$-VAE by providing a better trade-off between disentanglement and reconstruction quality. Moreover, we highlight the problems of a commonly used disentanglement metric and introduce a new metric that does not suffer from them.

arXiv commentShorter version appeared in Learning Disentangled Representations: From Perception to Control workshop at NIPS, 2017: https://sites.google.com/corp/view/disentanglenips2017

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Semi-Supervised Learning with Deep Generative Models
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Deep Convolutional Inverse Graphics Network
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Learning in Implicit Generative Models
• Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
• Variational inference for Monte Carlo objectives
• Disentangling Factors of Variation via Generative Entangling
• Learning Independent Features with Adversarial Nets for Non-linear ICA
• Learning the Irreducible Representations of Commutative Lie Groups
• Distribution Matching in Variational Inference

Cited by in corpus (213)

• Discovering physical concepts with neural networks
• Recent Advances in Autoencoder-Based Representation Learning
• Towards Out-Of-Distribution Generalization: A Survey
• Multi-Object Representation Learning with Iterative Variational Inference
• Disentangled Representation Learning in Cardiac Image Analysis
• MONet: Unsupervised Scene Decomposition and Representation
• Dynamical Variational Autoencoders: A Comprehensive Review
• InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization
• Swapping Autoencoder for Deep Image Manipulation
• Isolating Sources of Disentanglement in Variational Autoencoders
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Lifelong Generative Modeling
• Neural Style Transfer: A Review
• Learning Disentangled Representations in the Imaging Domain
• Weakly-Supervised Disentanglement Without Compromises
• Information Constraints on Auto-Encoding Variational Bayes
• Overlearning Reveals Sensitive Attributes
• Factorizable Graph Convolutional Networks
• Weakly Supervised Disentanglement with Guarantees
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Alleviating Structural Distribution Shift in Graph Anomaly Detection
• Lifelong Teacher-Student Network Learning
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Unsupervised Object Segmentation by Redrawing
• Learning Disentangled Joint Continuous and Discrete Representations
• Disentangling Disentanglement in Variational Autoencoders
• ControlVAE: Controllable Variational Autoencoder
• Disentangling Factors of Variation Using Few Labels
• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• Contrastive Variational Autoencoder Enhances Salient Features
• Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations
• On Disentangled Representations Learned From Correlated Data
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• A Causal View on Robustness of Neural Networks
• Unsupervised Belief Representation Learning with Information-Theoretic Variational Graph Auto-Encoders
• A Tutorial on Deep Latent Variable Models of Natural Language
• Deep Music Analogy Via Latent Representation Disentanglement
• Domain-invariant Similarity Activation Map Contrastive Learning for Retrieval-based Long-term Visual Localization
• Quantum device fine-tuning using unsupervised embedding learning
• Auto-Encoding Total Correlation Explanation
• Diffusion Variational Autoencoders
• Discrete and continuous representations and processing in deep learning: Looking forward
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Debiasing Learning for Membership Inference Attacks Against Recommender Systems
• Deep active inference agents using Monte-Carlo methods
• Data-Driven Approach to Encoding and Decoding 3-D Crystal Structures
• Operationally meaningful representations of physical systems in neural networks
• Music Style Transfer: A Position Paper
• Relevance Factor VAE: Learning and Identifying Disentangled Factors
• Decompose to manipulate: Manipulable Object Synthesis in 3D Medical Images with Structured Image Decomposition
• Towards causal generative scene models via competition of experts
• Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness
• Learning Disentangled Representations with Reference-Based Variational Autoencoders
• Image-to-Image Translation: Methods and Applications
• Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning
• Towards Visually Explaining Variational Autoencoders
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• Generative Latent Flow
• Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data
• Desiderata for Representation Learning: A Causal Perspective
• A Spectral Regularizer for Unsupervised Disentanglement
• Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning
• Semi-Supervised Disentangled Framework for Transferable Named Entity Recognition
• A New Multi-vehicle Trajectory Generator to Simulate Vehicle-to-Vehicle Encounters
• Jointly De-biasing Face Recognition and Demographic Attribute Estimation
• Generative causal explanations of black-box classifiers
• Semi-Supervised Generative Modeling for Controllable Speech Synthesis
• Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems
• Semi-Supervised StyleGAN for Disentanglement Learning
• Dual Swap Disentangling
• Multi-Scale and Multi-Layer Contrastive Learning for Domain Generalization
• RG-Flow: A hierarchical and explainable flow model based on renormalization group and sparse prior
• Elastic-InfoGAN: Unsupervised Disentangled Representation Learning in Class-Imbalanced Data
• Generating Tertiary Protein Structures via an Interpretative Variational Autoencoder
• Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement
• Theory and Evaluation Metrics for Learning Disentangled Representations
• An Identifiable Double VAE For Disentangled Representations
• Competitive Training of Mixtures of Independent Deep Generative Models
• Learning-Aided Physical Layer Attacks Against Multicarrier Communications in IoT
• Learning and controlling the source-filter representation of speech with a variational autoencoder
• Understanding Instance-based Interpretability of Variational Auto-Encoders
• Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images
• Generating Semantic Adversarial Examples via Feature Manipulation
• On the Transfer of Disentangled Representations in Realistic Settings
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• Guided Variational Autoencoder for Disentanglement Learning
• Semantics Disentangling for Generalized Zero-Shot Learning
• Learning Group Structure and Disentangled Representations of Dynamical Environments
• Disentangled Variational Representation for Heterogeneous Face Recognition
• Demystifying Inter-Class Disentanglement
• Regularized Autoencoders via Relaxed Injective Probability Flow
• Generative Neurosymbolic Machines
• Evaluating the Disentanglement of Deep Generative Models through Manifold Topology
• InfoCatVAE: Representation Learning with Categorical Variational Autoencoders
• Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning
• S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation
• Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations
• Disentangling and Learning Robust Representations with Natural Clustering
• On the Fairness of Disentangled Representations
• Human-interpretable model explainability on high-dimensional data
• Improving ClusterGAN Using Self-Augmented Information Maximization of Disentangling Latent Spaces
• Capturing Label Characteristics in VAEs
• CoDeGAN: Contrastive Disentanglement for Generative Adversarial Network
• Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglement
• Commutative Lie Group VAE for Disentanglement Learning
• Jigsaw-VAE: Towards Balancing Features in Variational Autoencoders
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case
• Learning Temporally Causal Latent Processes from General Temporal Data
• Improving VAEs' Robustness to Adversarial Attack
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Independence Promoted Graph Disentangled Networks
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• Generalization and Robustness Implications in Object-Centric Learning
• Measuring the Biases and Effectiveness of Content-Style Disentanglement
• Data-efficient visuomotor policy training using reinforcement learning and generative models
• Disentangling by Subspace Diffusion
• Bidirectional Generative Modeling Using Adversarial Gradient Estimation
• A Plug-in Method for Representation Factorization in Connectionist Models
• Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication
• Hyperprior Induced Unsupervised Disentanglement of Latent Representations
• Towards Principled Disentanglement for Domain Generalization
• Improving Disentangled Representation Learning with the Beta Bernoulli Process
• Unsupervised and interpretable scene discovery with Discrete-Attend-Infer-Repeat
• A Fine-Grained Analysis on Distribution Shift
• Disentangling to Cluster: Gaussian Mixture Variational Ladder Autoencoders
• Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip
• Mode matching in GANs through latent space learning and inversion
• Unpaired Image-to-Image Translation via Latent Energy Transport
• Semi-supervised Disentanglement with Independent Vector Variational Autoencoders
• Training Interpretable Convolutional Neural Networks by Differentiating Class-specific Filters
• DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability
• Edge-Enhanced Global Disentangled Graph Neural Network for Sequential Recommendation
• Tuning-Free Disentanglement via Projection
• Video Reenactment as Inductive Bias for Content-Motion Disentanglement
• Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model
• Representation Disentanglement for Multi-task Learning with application to Fetal Ultrasound
• Discond-VAE: Disentangling Continuous Factors from the Discrete
• Achieving Explainability for Plant Disease Classification with Disentangled Variational Autoencoders
• Identifying through Flows for Recovering Latent Representations
• LAVAE: Disentangling Location and Appearance
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling
• How to Not Measure Disentanglement
• Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders
• Fine-grained Sentiment Controlled Text Generation
• Product Manifold Learning
• Linear Disentangled Representations and Unsupervised Action Estimation
• Zero-shot Synthesis with Group-Supervised Learning
• Random Polytope Descriptors
• Multi-type Disentanglement without Adversarial Training
• Learning disentangled representations via product manifold projection
• OIAD: One-for-all Image Anomaly Detection with Disentanglement Learning
• Double cycle-consistent generative adversarial network for unsupervised conditional generation
• Rate-Regularization and Generalization in VAEs
• Contextual Joint Factor Acoustic Embeddings
• Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
• Bayes-Factor-VAE: Hierarchical Bayesian Deep Auto-Encoder Models for Factor Disentanglement
• Algorithm Fairness in AI for Medicine and Healthcare
• EditVAE: Unsupervised Part-Aware Controllable 3D Point Cloud Shape Generation
• GeomCA: Geometric Evaluation of Data Representations
• Learning Diverse Representations for Fast Adaptation to Distribution Shift
• Decoupling Global and Local Representations via Invertible Generative Flows
• Disentanglement Analysis with Partial Information Decomposition
• MaskAAE: Latent space optimization for Adversarial Auto-Encoders
• No Representation without Transformation
• Mixture Representation Learning with Coupled Autoencoders
• Revisiting Factorizing Aggregated Posterior in Learning Disentangled Representations
• Rethinking Content and Style: Exploring Bias for Unsupervised Disentanglement
• Inverse Learning of Symmetries
• Not All Features Are Equal: Feature Leveling Deep Neural Networks for Better Interpretation
• Defending Adversarial Attacks via Semantic Feature Manipulation
• Learning Bijective Feature Maps for Linear ICA
• Learning Conditional Invariance through Cycle Consistency
• Capturing Actionable Dynamics with Structured Latent Ordinary Differential Equations
• Disentanglement Learning via Topology
• Set-Based Face Recognition Beyond Disentanglement: Burstiness Suppression With Variance Vocabulary
• A Factorial Mixture Prior for Compositional Deep Generative Models
• Disentangling semantic features of macromolecules in Cryo-Electron Tomography
• Fair Representation Learning using Interpolation Enabled Disentanglement
• The Pitfall of More Powerful Autoencoders in Lidar-Based Navigation
• Geodesics in fibered latent spaces: A geometric approach to learning correspondences between conditions
• From Rain Generation to Rain Removal
• Benefiting Deep Latent Variable Models via Learning the Prior and Removing Latent Regularization
• Factorized Deep Generative Models for Trajectory Generation with Spatiotemporal-Validity Constraints
• IID-GAN: an IID Sampling Perspective for Regularizing Mode Collapse
• Learning Calibratable Policies using Programmatic Style-Consistency
• Adversarial Canonical Correlation Analysis
• Explaining Representation by Mutual Information
• Disentangling Action Sequences: Discovering Correlated Samples
• Group Equivariant Subsampling
• DAReN: A Collaborative Approach Towards Reasoning And Disentangling
• Decoder-free Robustness Disentanglement without (Additional) Supervision
• Unsupervised Program Synthesis for Images By Sampling Without Replacement
• DEFT: Distilling Entangled Factors by Preventing Information Diffusion
• Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders
• Bundle Networks: Fiber Bundles, Local Trivializations, and a Generative Approach to Exploring Many-to-one Maps
• Illiterate DALL-E Learns to Compose
• A Survey on Adversarial Image Synthesis
• The pursuit of beauty: Converting image labels to meaningful vectors
• Learning disentangled representation for classical models
• WiSE-ALE: Wide Sample Estimator for Approximate Latent Embedding
• Controllable Recommenders using Deep Generative Models and Disentanglement
• Group-disentangled Representation Learning with Weakly-Supervised Regularization
• Disentangling Representations of Text by Masking Transformers
• NeurIPS 2019 Disentanglement Challenge: Improved Disentanglement through Learned Aggregation of Convolutional Feature Maps
• Quantifying and Learning Linear Symmetry-Based Disentanglement
• Longitudinal Self-Supervised Learning
• Nested Scale Editing for Conditional Image Synthesis
• Disentangled Unsupervised Image Translation via Restricted Information Flow
• Towards Better Understanding of Disentangled Representations via Mutual Information
• Kullback-Leibler Divergence-Based Out-of-Distribution Detection with Flow-Based Generative Models
• SepVAE: a contrastive VAE to separate pathological patterns from healthy ones