Information bottleneck through variational glasses

Abstract

Information bottleneck (IB) principle [1] has become an important element in information-theoretic analysis of deep models. Many state-of-the-art generative models of both Variational Autoencoder (VAE) [2; 3] and Generative Adversarial Networks (GAN) [4] families use various bounds on mutual information terms to introduce certain regularization constraints [5; 6; 7; 8; 9; 10]. Accordingly, the main difference between these models consists in add regularization constraints and targeted objectives. In this work, we will consider the IB framework for three classes of models that include supervised, unsupervised and adversarial generative models. We will apply a variational decomposition leading a common structure and allowing easily establish connections between these models and analyze underlying assumptions. Based on these results, we focus our analysis on unsupervised setup and reconsider the VAE family. In particular, we present a new interpretation of VAE family based on the IB framework using a direct decomposition of mutual information terms and show some interesting connections to existing methods such as VAE [2; 3], beta-VAE [11], AAE [12], InfoVAE [5] and VAE/GAN [13]. Instead of adding regularization constraints to an evidence lower bound (ELBO) [2; 3], which itself is a lower bound, we show that many known methods can be considered as a product of variational decomposition of mutual information terms in the IB framework. The proposed decomposition might also contribute to the interpretability of generative models of both VAE and GAN families and create a new insights to a generative compression [14; 15; 16; 17]. It can also be of interest for the analysis of novelty detection based on one-class classifiers [18] with the IB based discriminators.

References in corpus (12)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• The information bottleneck method
• Autoencoding beyond pixels using a learned similarity metric
• InfoVAE: Information Maximizing Variational Autoencoders
• Deep Variational Information Bottleneck
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Learning in Implicit Generative Models
• Generative Probabilistic Novelty Detection with Adversarial Autoencoders
• Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff
• Fixing a Broken ELBO
• The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models

Cited by in corpus (7)

• Getting High: High Fidelity Simulation of High Granularity Calorimeters with High Speed
• Decoding Photons: Physics in the Latent Space of a BIB-AE Generative Network
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Variational Autoencoders for Jet Simulation
• TURBO: The Swiss Knife of Auto-Encoders
• Machine learning attack on copy detection patterns: are 1x1 patterns cloneable?
• Blocked and Hierarchical Disentangled Representation From Information Theory Perspective