ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA

Abstract

We consider the identifiability theory of probabilistic models and establish sufficient conditions under which the representations learned by a very broad family of conditional energy-based models are unique in function space, up to a simple transformation. In our model family, the energy function is the dot-product between two feature extractors, one for the dependent variable, and one for the conditioning variable. We show that under mild conditions, the features are unique up to scaling and permutation. Our results extend recent developments in nonlinear ICA, and in fact, they lead to an important generalization of ICA models. In particular, we show that our model can be used for the estimation of the components in the framework of Independently Modulated Component Analysis (IMCA), a new generalization of nonlinear ICA that relaxes the independence assumption. A thorough empirical study shows that representations learned by our model from real-world image datasets are identifiable, and improve performance in transfer learning and semi-supervised learning tasks.

arXiv commentAccepted for publication at NeurIPS 2020

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Generative Modeling by Estimating Gradients of the Data Distribution
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
• Isolating Sources of Disentanglement in Variational Autoencoders
• On gradient regularizers for MMD GANs
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• Deep Energy Estimator Networks
• Few-shot Domain Adaptation by Causal Mechanism Transfer
• Causal Discovery with General Non-Linear Relationships Using Non-Linear ICA
• A Unified Probabilistic Model for Learning Latent Factors and Their Connectivities from High-Dimensional Data

Cited by in corpus (10)

• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• On Linear Identifiability of Learned Representations
• How to Train Your Energy-Based Model for Regression
• Unsupervised Learning of Compositional Energy Concepts
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Causal Hidden Markov Model for Time Series Disease Forecasting
• $\beta$-Intact-VAE: Identifying and Estimating Causal Effects under Limited Overlap
• Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learning