Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)

Abstract

A central question of representation learning asks under which conditions it is possible to reconstruct the true latent variables of an arbitrarily complex generative process. Recent breakthrough work by Khemakhem et al. (2019) on nonlinear ICA has answered this question for a broad class of conditional generative processes. We extend this important result in a direction relevant for application to real-world data. First, we generalize the theory to the case of unknown intrinsic problem dimension and prove that in some special (but not very restrictive) cases, informative latent variables will be automatically separated from noise by an estimating model. Furthermore, the recovered informative latent variables will be in one-to-one correspondence with the true latent variables of the generating process, up to a trivial component-wise transformation. Second, we introduce a modification of the RealNVP invertible neural network architecture (Dinh et al. (2016)) which is particularly suitable for this type of problem: the General Incompressible-flow Network (GIN). Experiments on artificial data and EMNIST demonstrate that theoretical predictions are indeed verified in practice. In particular, we provide a detailed set of exactly 22 informative latent variables extracted from EMNIST.

arXiv comment23 pages, 15 figures, ICLR 2020

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• EMNIST: an extension of MNIST to handwritten letters
• Guided Image Generation with Conditional Invertible Neural Networks
• Composing graphical models with neural networks for structured representations and fast inference
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework
• i-RevNet: Deep Invertible Networks
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning

Cited by in corpus (21)

• Weakly-Supervised Disentanglement Without Compromises
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE
• Unshuffling Data for Improved Generalization
• ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series
• On Linear Identifiability of Learned Representations
• Learning and controlling the source-filter representation of speech with a variational autoencoder
• On the Transfer of Disentangled Representations in Realistic Settings
• Intact-VAE: Estimating Treatment Effects under Unobserved Confounding
• Learning Temporally Causal Latent Processes from General Temporal Data
• Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglement
• Temperature-steerable flows
• InFlow: Robust outlier detection utilizing Normalizing Flows
• Learning Non-linear Wavelet Transformation via Normalizing Flow
• iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression
• Time Series Source Separation with Slow Flows
• Bundle Networks: Fiber Bundles, Local Trivializations, and a Generative Approach to Exploring Many-to-one Maps
• Conditional Invertible Neural Networks for Diverse Image-to-Image Translation
• DEFT: Distilling Entangled Factors by Preventing Information Diffusion