Learning Independent Features with Adversarial Nets for Non-linear ICA

Abstract

Reliable measures of statistical dependence could be useful tools for learning independent features and performing tasks like source separation using Independent Component Analysis (ICA). Unfortunately, many of such measures, like the mutual information, are hard to estimate and optimize directly. We propose to learn independent features with adversarial objectives which optimize such measures implicitly. These objectives compare samples from the joint distribution and the product of the marginals without the need to compute any probability densities. We also propose two methods for obtaining samples from the product of the marginals using either a simple resampling trick or a separate parametric distribution. Our experiments show that this strategy can easily be applied to different types of model architectures and solve both linear and non-linear ICA problems.

A preliminary version of this work was presented at the ICML 2017 workshop on implicit models

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• Improved Training of Wasserstein GANs
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Least Squares Generative Adversarial Networks
• Boundary-Seeking Generative Adversarial Networks
• Invariant Representations for Noisy Speech Recognition

Cited by in corpus (25)

• Disentangling by Factorising
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework
• Maximum Entropy Generators for Energy-Based Models
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning
• Self-Supervised Relational Reasoning for Representation Learning
• Relevance Factor VAE: Learning and Identifying Disentangled Factors
• Learning Disentangled Representations via Mutual Information Estimation
• Innovations Autoencoder and its Application in One-class Anomalous Sequence Detection
• Biphasic Learning of GANs for High-Resolution Image-to-Image Translation
• A Critical View of the Structural Causal Model
• Training Generative Adversarial Networks from Incomplete Observations using Factorised Discriminators
• Bayes-Factor-VAE: Hierarchical Bayesian Deep Auto-Encoder Models for Factor Disentanglement
• Learning Diverse Representations for Fast Adaptation to Distribution Shift
• Learning Segmentation Masks with the Independence Prior
• Mutual Information Regularized Identity-aware Facial ExpressionRecognition in Compressed Video
• No Representation without Transformation
• Free Component Analysis: Theory, Algorithms & Applications
• Towards Better Understanding of Disentangled Representations via Mutual Information
• Label-Based Diversity Measure Among Hidden Units of Deep Neural Networks: A Regularization Method
• Analyzing the benefits of communication channels between deep learning models
• Learning Disentangled Latent Factors from Paired Data in Cross-Modal Retrieval: An Implicit Identifiable VAE Approach
• Weakly Supervised Recovery of Semantic Attributes
• A Normative and Biologically Plausible Algorithm for Independent Component Analysis
• WICA: nonlinear weighted ICA
• AID-Purifier: A Light Auxiliary Network for Boosting Adversarial Defense