Variational Autoencoders and Nonlinear ICA: A Unifying Framework

Abstract

The framework of variational autoencoders allows us to efficiently learn deep latent-variable models, such that the model's marginal distribution over observed variables fits the data. Often, we're interested in going a step further, and want to approximate the true joint distribution over observed and latent variables, including the true prior and posterior distributions over latent variables. This is known to be generally impossible due to unidentifiability of the model. We address this issue by showing that for a broad family of deep latent-variable models, identification of the true joint distribution over observed and latent variables is actually possible up to very simple transformations, thus achieving a principled and powerful form of disentanglement. Our result requires a factorized prior distribution over the latent variables that is conditioned on an additionally observed variable, such as a class label or almost any other observation. We build on recent developments in nonlinear ICA, which we extend to the case with noisy, undercomplete or discrete observations, integrated in a maximum likelihood framework. The result also trivially contains identifiable flow-based generative models as a special case.

Accepted for publication at AISTATS 2020. This is a slightly updated version of the published manuscript; see Corrigendum at the end of the paper

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• NICE: Non-linear Independent Components Estimation
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• Towards a Definition of Disentangled Representations
• Understanding disentangling in $\beta$-VAE
• Importance Weighted Autoencoders
• Improving Variational Inference with Inverse Autoregressive Flow
• Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Density Estimation in Infinite Dimensional Exponential Families
• Learning Independent Features with Adversarial Nets for Non-linear ICA
• Disentangling Disentanglement in Variational Autoencoders

Cited by in corpus (20)

• Learning Disentangled Representations in the Imaging Domain
• Weakly-Supervised Disentanglement Without Compromises
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• State estimation with limited sensors -- A deep learning based approach
• Interpretable brain age prediction using linear latent variable models of functional connectivity
• Information Maximization Clustering via Multi-View Self-Labelling
• Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA
• Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series
• Deep Generative Imputation Model for Missing Not At Random Data
• On Linear Identifiability of Learned Representations
• Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
• Dual Adversarial Inference for Text-to-Image Synthesis
• DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability
• Amortized backward variational inference in nonlinear state-space models
• Time Series Source Separation with Slow Flows
• WICA: nonlinear weighted ICA
• SepVAE: a contrastive VAE to separate pathological patterns from healthy ones
• A Deeper Look at the Unsupervised Learning of Disentangled Representations in $\beta$-VAE from the Perspective of Core Object Recognition