Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning

Abstract

Nonlinear ICA is a fundamental problem for unsupervised representation learning, emphasizing the capacity to recover the underlying latent variables generating the data (i.e., identifiability). Recently, the very first identifiability proofs for nonlinear ICA have been proposed, leveraging the temporal structure of the independent components. Here, we propose a general framework for nonlinear ICA, which, as a special case, can make use of temporal structure. It is based on augmenting the data by an auxiliary variable, such as the time index, the history of the time series, or any other available information. We propose to learn nonlinear ICA by discriminating between true augmented data, or data in which the auxiliary variable has been randomized. This enables the framework to be implemented algorithmically through logistic regression, possibly in a neural network. We provide a comprehensive proof of the identifiability of the model as well as the consistency of our estimation method. The approach not only provides a general theoretical framework combining and generalizing previously proposed nonlinear ICA models and algorithms, but also brings practical advantages.

arXiv commentCamera-ready version of article accepted for AISTATS2019

References in corpus (4)

• Representation Learning with Contrastive Predictive Coding
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
• Learning Independent Features with Adversarial Nets for Non-linear ICA

Cited by in corpus (14)

• A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
• Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Series
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• On Linear Identifiability of Learned Representations
• Explicit Disentanglement of Appearance and Perspective in Generative Models
• Unsupervised Learning of Compositional Energy Concepts
• Modelling multivariate spatio-temporal data with identifiable variational autoencoders
• Nonlinear blind source separation exploiting spatial nonstationarity
• Relative gradient optimization of the Jacobian term in unsupervised deep learning
• Identifying through Flows for Recovering Latent Representations
• Free Component Analysis: Theory, Algorithms & Applications
• Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments