Relevant sparse codes with variational information bottleneck

Abstract

In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y, while constraining the information encoded about the original data, X. Unfortunately however, the IB method is computationally demanding when data are high-dimensional and/or non-gaussian. Here we propose an approximate variational scheme for maximizing a lower bound on the IB objective, analogous to variational EM. Using this method, we derive an IB algorithm to recover features that are both relevant and sparse. Finally, we demonstrate how kernelized versions of the algorithm can be used to address a broad range of problems with non-linear relation between X and Y.

References in corpus (1)

• The Information Bottleneck EM Algorithm

Cited by in corpus (24)

• Deep Variational Information Bottleneck
• Nonlinear Information Bottleneck
• Entropy and mutual information in models of deep neural networks
• Meta reinforcement learning as task inference
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Significance-aware Information Bottleneck for Domain Adaptive Semantic Segmentation
• The Convex Information Bottleneck Lagrangian
• The Deep Kernelized Autoencoder
• Learning Optimal Representations with the Decodable Information Bottleneck
• Caveats for information bottleneck in deterministic scenarios
• Learnability for the Information Bottleneck
• Compression-Based Regularization with an Application to Multi-Task Learning
• Bottleneck Problems: Information and Estimation-Theoretic View
• Pareto-optimal data compression for binary classification tasks
• Gaussian Lower Bound for the Information Bottleneck Limit
• Information Theoretic Meta Learning with Gaussian Processes
• IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL
• Physics-constrained, data-driven discovery of coarse-grained dynamics
• Mutual Information State Intrinsic Control
• Instance-Aware Graph Convolutional Network for Multi-Label Classification
• Variational Information Bottleneck for Effective Low-resource Audio Classification
• Attentional Bottleneck: Towards an Interpretable Deep Driving Network
• Kernelized Hashcode Representations for Relation Extraction
• Modeling Psychotherapy Dialogues with Kernelized Hashcode Representations: A Nonparametric Information-Theoretic Approach