Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA

Abstract

Nonlinear independent component analysis (ICA) provides an appealing framework for unsupervised feature learning, but the models proposed so far are not identifiable. Here, we first propose a new intuitive principle of unsupervised deep learning from time series which uses the nonstationary structure of the data. Our learning principle, time-contrastive learning (TCL), finds a representation which allows optimal discrimination of time segments (windows). Surprisingly, we show how TCL can be related to a nonlinear ICA model, when ICA is redefined to include temporal nonstationarities. In particular, we show that TCL combined with linear ICA estimates the nonlinear ICA model up to point-wise transformations of the sources, and this solution is unique --- thus providing the first identifiability result for nonlinear ICA which is rigorous, constructive, as well as very general.

References in corpus (3)

• NICE: Non-linear Independent Components Estimation
• Unsupervised Feature Learning from Temporal Data
• Likelihood-free inference via classification

Cited by in corpus (54)

• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Neural Image Compression for Gigapixel Histopathology Image Analysis
• Unsupervised Scalable Representation Learning for Multivariate Time Series
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Weakly-Supervised Disentanglement Without Compromises
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Self-supervised learning through the eyes of a child
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Contrastive Learning with Adversarial Examples
• Colorization as a Proxy Task for Visual Understanding
• Disentangling Factors of Variation Using Few Labels
• Deep Energy Estimator Networks
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning
• On Disentangled Representations Learned From Correlated Data
• Few-shot Domain Adaptation by Causal Mechanism Transfer
• Deep Unsupervised Learning of Visual Similarities
• CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients
• ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA
• Deep Reinforcement and InfoMax Learning
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder
• Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series
• Contrastive learning, multi-view redundancy, and linear models
• Nonlinear Multiview Analysis: Identifiability and Neural Network-assisted Implementation
• Learning Nonlinear Mixtures: Identifiability and Algorithm
• On the Transfer of Disentangled Representations in Realistic Settings
• Latent Causal Invariant Model
• Robustifying Independent Component Analysis by Adjusting for Group-Wise Stationary Noise
• On the Fairness of Disentangled Representations
• Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA
• Modelling multivariate spatio-temporal data with identifiable variational autoencoders
• Nonlinear blind source separation exploiting spatial nonstationarity
• Learning Temporally Causal Latent Processes from General Temporal Data
• Intact-VAE: Estimating Treatment Effects under Unobserved Confounding
• The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICA
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Telescoping Density-Ratio Estimation
• Independent Innovation Analysis for Nonlinear Vector Autoregressive Process
• Relative gradient optimization of the Jacobian term in unsupervised deep learning
• Causal Mosaic: Cause-Effect Inference via Nonlinear ICA and Ensemble Method
• Identifying through Flows for Recovering Latent Representations
• Discovering Predictable Latent Factors for Time Series Forecasting
• Learning disentangled representations via product manifold projection
• Robust contrastive learning and nonlinear ICA in the presence of outliers
• Independent mechanism analysis, a new concept?
• Free Component Analysis: Theory, Algorithms & Applications
• Unified estimation framework for unnormalized models with statistical efficiency
• Estimation of Non-Normalized Mixture Models and Clustering Using Deep Representation
• Disentangling Video with Independent Prediction
• Identifiable Energy-based Representations: An Application to Estimating Heterogeneous Causal Effects
• Discriminative Clustering with Representation Learning with any Ratio of Labeled to Unlabeled Data
• WICA: nonlinear weighted ICA
• Minimax Analysis for Inverse Risk in Nonparametric Planer Invertible Regression