PRI-VAE: Principle-of-Relevant-Information Variational Autoencoders

Abstract

Although substantial efforts have been made to learn disentangled representations under the variational autoencoder (VAE) framework, the fundamental properties to the dynamics of learning of most VAE models still remain unknown and under-investigated. In this work, we first propose a novel learning objective, termed the principle-of-relevant-information variational autoencoder (PRI-VAE), to learn disentangled representations. We then present an information-theoretic perspective to analyze existing VAE models by inspecting the evolution of some critical information-theoretic quantities across training epochs. Our observations unveil some fundamental properties associated with VAEs. Empirical results also demonstrate the effectiveness of PRI-VAE on four benchmark data sets.

References in corpus (9)

• Auto-Encoding Variational Bayes
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Estimating Mutual Information
• Opening the Black Box of Deep Neural Networks via Information
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Recent Advances in Autoencoder-Based Representation Learning
• Understanding disentangling in $\beta$-VAE
• Structured Disentangled Representations
• Adaptive Estimators Show Information Compression in Deep Neural Networks