Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations

Abstract

The key idea behind the unsupervised learning of disentangled representations is that real-world data is generated by a few explanatory factors of variation which can be recovered by unsupervised learning algorithms. In this paper, we provide a sober look at recent progress in the field and challenge some common assumptions. We first theoretically show that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data. Then, we train more than 12000 models covering most prominent methods and evaluation metrics in a reproducible large-scale experimental study on seven different data sets. We observe that while the different methods successfully enforce properties ``encouraged'' by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision. Furthermore, increased disentanglement does not seem to lead to a decreased sample complexity of learning for downstream tasks. Our results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision, investigate concrete benefits of enforcing disentanglement of the learned representations, and consider a reproducible experimental setup covering several data sets.

Cited by in corpus (261)

• Recent Advances in Autoencoder-Based Representation Learning
• Towards Out-Of-Distribution Generalization: A Survey
• Multi-Object Representation Learning with Iterative Variational Inference
• How Does Learning Rate Decay Help Modern Neural Networks?
• Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework
• Causality for Machine Learning
• Deep Learning and Knowledge-Based Methods for Computer Aided Molecular Design -- Toward a Unified Approach: State-of-the-Art and Future Directions
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Learning Disentangled Representations in the Imaging Domain
• Weakly-Supervised Disentanglement Without Compromises
• Diagnosing and Enhancing VAE Models
• DIVA: Domain Invariant Variational Autoencoders
• DDSP: Differentiable Digital Signal Processing
• A causal view of compositional zero-shot recognition
• Weakly Supervised Disentanglement with Guarantees
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Disentangling Disentanglement in Variational Autoencoders
• On Completeness-aware Concept-Based Explanations in Deep Neural Networks
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Disentangling Factors of Variation Using Few Labels
• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• On Disentangled Representations Learned From Correlated Data
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• Discrete and continuous representations and processing in deep learning: Looking forward
• Convolutional Generation of Textured 3D Meshes
• Meta-learning an Intermediate Representation for Few-shot Block-wise Prediction of Landslide Susceptibility
• Contrastive Learning Inverts the Data Generating Process
• Counterfactuals uncover the modular structure of deep generative models
• Toward a Better Monitoring Statistic for Profile Monitoring via Variational Autoencoders
• Improving Zero-shot Voice Style Transfer via Disentangled Representation Learning
• Unsupervised State Representation Learning in Atari
• ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA
• Nonlinear Invariant Risk Minimization: A Causal Approach
• Perfect density models cannot guarantee anomaly detection
• SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling
• Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness
• Towards causal generative scene models via competition of experts
• Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA
• Evaluating Weakly Supervised Object Localization Methods Right
• Learning Interpretable Representation for Controllable Polyphonic Music Generation
• Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning
• Neuromorphic Visual Scene Understanding with Resonator Networks
• Impossibility Results in AI: A Survey
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• DISP6D: Disentangled Implicit Shape and Pose Learning for Scalable 6D Pose Estimation
• What Matters for Adversarial Imitation Learning?
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder
• Desiderata for Representation Learning: A Causal Perspective
• Exploiting auto-encoders and segmentation methods for middle-level explanations of image classification systems
• Disentangled Recurrent Wasserstein Autoencoder
• Face Anti-Spoofing Via Disentangled Representation Learning
• Semi-Supervised Disentangled Framework for Transferable Named Entity Recognition
• Jointly De-biasing Face Recognition and Demographic Attribute Estimation
• Disentangling User Interest and Conformity for Recommendation with Causal Embedding
• Symmetry-Based Disentangled Representation Learning requires Interaction with Environments
• Disentangled State Space Representations
• RG-Flow: A hierarchical and explainable flow model based on renormalization group and sparse prior
• Contrastively Disentangled Sequential Variational Autoencoder
• Explicit Disentanglement of Appearance and Perspective in Generative Models
• Controlling for Confounders in Multimodal Emotion Classification via Adversarial Learning
• Effect of The Latent Structure on Clustering with GANs
• Competitive Training of Mixtures of Independent Deep Generative Models
• Treatment effect estimation with disentangled latent factors
• An Identifiable Double VAE For Disentangled Representations
• Visual Compositional Learning for Human-Object Interaction Detection
• Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Learning and controlling the source-filter representation of speech with a variational autoencoder
• Feature Unlearning for Pre-trained GANs and VAEs
• PDE-Driven Spatiotemporal Disentanglement
• Learning Group Structure and Disentangled Representations of Dynamical Environments
• On the Transfer of Disentangled Representations in Realistic Settings
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• Generative Neurosymbolic Machines
• AdaCoSeg: Adaptive Shape Co-Segmentation with Group Consistency Loss
• Demystifying Inter-Class Disentanglement
• Size-Invariant Graph Representations for Graph Classification Extrapolations
• Evaluating the Disentanglement of Deep Generative Models through Manifold Topology
• Triple Generative Adversarial Networks
• Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning
• Disentangled Representation Learning with Wasserstein Total Correlation
• S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation
• Spatial Dependency Networks: Neural Layers for Improved Generative Image Modeling
• HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
• Disentangled Representations from Non-Disentangled Models
• GeneDisco: A Benchmark for Experimental Design in Drug Discovery
• Causally motivated Shortcut Removal Using Auxiliary Labels
• Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations
• On the Fairness of Disentangled Representations
• FedDis: Disentangled Federated Learning for Unsupervised Brain Pathology Segmentation
• Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis
• Elucidating image-to-set prediction: An analysis of models, losses and datasets
• Structured Generative Models for Scene Understanding
• Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA
• Weakly Supervised Disentanglement by Pairwise Similarities
• Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglement
• Interactive Text-to-Speech System via Joint Style Analysis
• Multichannel-based learning for audio object extraction
• CoDeGAN: Contrastive Disentanglement for Generative Adversarial Network
• Capturing Label Characteristics in VAEs
• Learning Disentangled Expression Representations from Facial Images
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Learning Disentangled Representations for Time Series
• Improving VAEs' Robustness to Adversarial Attack
• Augmenting Variational Autoencoders with Sparse Labels: A Unified Framework for Unsupervised, Semi-(un)supervised, and Supervised Learning
• Towards Robust Metrics for Concept Representation Evaluation
• EigenGame: PCA as a Nash Equilibrium
• Learning Temporally Causal Latent Processes from General Temporal Data
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• A Critical Look at the Consistency of Causal Estimation With Deep Latent Variable Models
• On the Interpretability and Evaluation of Graph Representation Learning
• Multi-Facet Clustering Variational Autoencoders
• An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wild
• DynamicVAE: Decoupling Reconstruction Error and Disentangled Representation Learning
• A Plug-in Method for Representation Factorization in Connectionist Models
• Max-Affine Spline Insights into Deep Generative Networks
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Active World Model Learning with Progress Curiosity
• Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
• An Improved Semi-Supervised VAE for Learning Disentangled Representations
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• FaceController: Controllable Attribute Editing for Face in the Wild
• Generalization and Robustness Implications in Object-Centric Learning
• Disentangling by Subspace Diffusion
• Challenging common interpretability assumptions in feature attribution explanations
• Data-efficient visuomotor policy training using reinforcement learning and generative models
• Failure Modes of Variational Autoencoders and Their Effects on Downstream Tasks
• Measuring the Biases and Effectiveness of Content-Style Disentanglement
• Meta-Learning Dynamics Forecasting Using Task Inference
• Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
• Leveraging directed causal discovery to detect latent common causes
• Improving Disentangled Text Representation Learning with Information-Theoretic Guidance
• A Fine-Grained Analysis on Distribution Shift
• OOGAN: Disentangling GAN with One-Hot Sampling and Orthogonal Regularization
• Improving Disentangled Representation Learning with the Beta Bernoulli Process
• Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication
• On Disentanglement in Gaussian Process Variational Autoencoders
• Disentangling representations of retinal images with generative models
• Latent feature disentanglement for 3D meshes
• dMelodies: A Music Dataset for Disentanglement Learning
• Mode matching in GANs through latent space learning and inversion
• Edge-Enhanced Global Disentangled Graph Neural Network for Sequential Recommendation
• Video Reenactment as Inductive Bias for Content-Motion Disentanglement
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling
• Training Interpretable Convolutional Neural Networks by Differentiating Class-specific Filters
• Self-supervised Visual Reinforcement Learning with Object-centric Representations
• Semi-supervised Disentanglement with Independent Vector Variational Autoencoders
• PRI-VAE: Principle-of-Relevant-Information Variational Autoencoders
• How to Not Measure Disentanglement
• Identifying through Flows for Recovering Latent Representations
• Learning Disentangled Representations with Latent Variation Predictability
• LAVAE: Disentangling Location and Appearance
• Emergence of Numeric Concepts in Multi-Agent Autonomous Communication
• Adversarial Disentanglement with Grouped Observations
• Learning Disentangled Representations of Video with Missing Data
• Learning Sampling in Financial Statement Audits using Vector Quantised Autoencoder Neural Networks
• GeomCA: Geometric Evaluation of Data Representations
• Algorithm Fairness in AI for Medicine and Healthcare
• Linear Disentangled Representations and Unsupervised Action Estimation
• Rate-Regularization and Generalization in VAEs
• Simple, Scalable, and Stable Variational Deep Clustering
• Are standard Object Segmentation models sufficient for Learning Affordance Segmentation?
• Disentanglement with Hyperspherical Latent Spaces using Diffusion Variational Autoencoders
• Robust contrastive learning and nonlinear ICA in the presence of outliers
• Causal Hidden Markov Model for Time Series Disease Forecasting
• Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval
• STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech
• Hierarchical Variational Auto-Encoding for Unsupervised Domain Generalization
• Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective
• A Commentary on the Unsupervised Learning of Disentangled Representations
• Deep Anomaly Detection by Residual Adaptation
• On the Transferability of VAE Embeddings using Relational Knowledge with Semi-Supervision
• Towards a Theoretical Understanding of the Robustness of Variational Autoencoders
• SCNet: A Generalized Attention-based Model for Crack Fault Segmentation
• Learning disentangled representations via product manifold projection
• Bayes-Factor-VAE: Hierarchical Bayesian Deep Auto-Encoder Models for Factor Disentanglement
• EditVAE: Unsupervised Part-Aware Controllable 3D Point Cloud Shape Generation
• Transporting Causal Mechanisms for Unsupervised Domain Adaptation
• WeLa-VAE: Learning Alternative Disentangled Representations Using Weak Labels
• Generalization to Novel Objects using Prior Relational Knowledge
• Random Polytope Descriptors
• Don't miss the Mismatch: Investigating the Objective Function Mismatch for Unsupervised Representation Learning
• Product Manifold Learning
• Variation Network: Learning High-level Attributes for Controlled Input Manipulation
• Controllable Data Augmentation Through Deep Relighting
• Revisiting Factorizing Aggregated Posterior in Learning Disentangled Representations
• Learning to Infer Semantic Parameters for 3D Shape Editing
• Mixture Representation Learning with Coupled Autoencoders
• $\beta$-Annealed Variational Autoencoder for glitches
• Compositionality Through Language Transmission, using Artificial Neural Networks
• Learning from Demonstration with Weakly Supervised Disentanglement
• Blocked and Hierarchical Disentangled Representation From Information Theory Perspective
• Independent mechanism analysis, a new concept?
• Semi-Supervised Disentanglement of Class-Related and Class-Independent Factors in VAE
• StyleFusion: A Generative Model for Disentangling Spatial Segments
• Unsupervised Disentanglement of Linear-Encoded Facial Semantics
• Variance Constrained Autoencoding
• No Representation without Transformation
• Learning Invariances for Interpretability using Supervised VAE
• Learning Calibratable Policies using Programmatic Style-Consistency
• Unsupervised Representation Disentanglement of Text: An Evaluation on Synthetic Datasets
• Learning Conditional Invariance through Cycle Consistency
• Capturing Actionable Dynamics with Structured Latent Ordinary Differential Equations
• Controllable Gradient Item Retrieval
• Semi-supervised Meta-learning with Disentanglement for Domain-generalised Medical Image Segmentation
• Rethinking Content and Style: Exploring Bias for Unsupervised Disentanglement
• Inference-InfoGAN: Inference Independence via Embedding Orthogonal Basis Expansion
• Decoder-free Robustness Disentanglement without (Additional) Supervision
• Informative GANs via Structured Regularization of Optimal Transport
• Longitudinal Correlation Analysis for Decoding Multi-Modal Brain Development
• Towards Better Understanding of Disentangled Representations via Mutual Information
• A Proposal for Intelligent Agents with Episodic Memory
• Disentangling Action Sequences: Discovering Correlated Samples
• On the Limitations of Multimodal VAEs
• Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder
• Disentangling Representations of Text by Masking Transformers
• Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders
• On Controlled DeEntanglement for Natural Language Processing
• Learning Disentangled Latent Factors from Paired Data in Cross-Modal Retrieval: An Implicit Identifiable VAE Approach
• Semantic Network Interpretation
• On the Parameter Combinations That Matter and on Those That do Not
• Differential Similarity in Higher Dimensional Spaces: Theory and Applications
• Safer End-to-End Autonomous Driving via Conditional Imitation Learning and Command Augmentation
• Domain-Agnostic Learning with Anatomy-Consistent Embedding for Cross-Modality Liver Segmentation
• Divide and Rule: Recurrent Partitioned Network for Dynamic Processes
• Learning disentangled representation for classical models
• Noise Contrastive Variational Autoencoders
• TexRel: a Green Family of Datasets for Emergent Communications on Relations
• Weakly Supervised Recovery of Semantic Attributes
• Recursively Conditional Gaussian for Ordinal Unsupervised Domain Adaptation
• Variational Auto-Encoder Architectures that Excel at Causal Inference
• Label-Based Diversity Measure Among Hidden Units of Deep Neural Networks: A Regularization Method
• Disentangling Physical Parameters for Anomalous Sound Detection Under Domain Shifts
• Text Classification through Glyph-aware Disentangled Character Embedding and Semantic Sub-character Augmentation
• Generalizable and Explainable Dialogue Generation via Explicit Action Learning
• Encoding Causal Macrovariables
• Robust Disentanglement of a Few Factors at a Time
• Efficiently Disentangle Causal Representations
• Quantifying and Learning Linear Symmetry-Based Disentanglement
• Local Disentanglement in Variational Auto-Encoders Using Jacobian $L_1$ Regularization
• Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learning
• Adaptive Compression of the Latent Space in Variational Autoencoders
• Group-disentangled Representation Learning with Weakly-Supervised Regularization
• Leveraging Hidden Structure in Self-Supervised Learning
• Kullback-Leibler Divergence-Based Out-of-Distribution Detection with Flow-Based Generative Models
• Deep Variational Luenberger-type Observer for Stochastic Video Prediction
• Differentiable Disentanglement Filter: an Application Agnostic Core Concept Discovery Probe
• A Deeper Look at the Unsupervised Learning of Disentangled Representations in $\beta$-VAE from the Perspective of Core Object Recognition
• Unsupervised Program Synthesis for Images By Sampling Without Replacement
• DEFT: Distilling Entangled Factors by Preventing Information Diffusion
• Proceedings of the First Workshop on Weakly Supervised Learning (WeaSuL)
• Explainability Requires Interactivity
• Group Equivariant Subsampling
• Deep Automodulators
• Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders