Information Constraints on Auto-Encoding Variational Bayes

Abstract

Parameterizing the approximate posterior of a generative model with neural networks has become a common theme in recent machine learning research. While providing appealing flexibility, this approach makes it difficult to impose or assess structural constraints such as conditional independence. We propose a framework for learning representations that relies on Auto-Encoding Variational Bayes and whose search space is constrained via kernel-based measures of independence. In particular, our method employs the $d$-variable Hilbert-Schmidt Independence Criterion (dHSIC) to enforce independence between the latent representations and arbitrary nuisance factors. We show how to apply this method to a range of problems, including the problems of learning invariant representations and the learning of interpretable representations. We also present a full-fledged application to single-cell RNA sequencing (scRNA-seq). In this setting the biological signal is mixed in complex ways with sequencing errors and sampling effects. We show that our method out-performs the state-of-the-art in this domain.

References in corpus (9)

• Auto-Encoding Variational Bayes
• Semi-Supervised Learning with Deep Generative Models
• Supervised Dictionary Learning
• Disentangling by Factorising
• Adversarial Autoencoders
• Understanding disentangling in $\beta$-VAE
• Composing graphical models with neural networks for structured representations and fast inference
• Characteristic and Universal Tensor Product Kernels
• An Adaptive Test of Independence with Analytic Kernel Embeddings

Cited by in corpus (10)

• Recent Advances in Autoencoder-Based Representation Learning
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Generating Tertiary Protein Structures via an Interpretative Variational Autoencoder
• Unified Adversarial Invariance
• Knowledge-Based Regularization in Generative Modeling
• Disentangling and Learning Robust Representations with Natural Clustering
• Contrastive Mixture of Posteriors for Counterfactual Inference, Data Integration and Fairness
• Variational Auto-Encoder Architectures that Excel at Causal Inference
• Towards Low-Resource StarGAN Voice Conversion using Weight Adaptive Instance Normalization
• Disentangling Action Sequences: Discovering Correlated Samples