Nonlinear Information Bottleneck

Abstract

Information bottleneck (IB) is a technique for extracting information in one random variable $X$ that is relevant for predicting another random variable $Y$. IB works by encoding $X$ in a compressed "bottleneck" random variable $M$ from which $Y$ can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete $X$ and $Y$ with small state spaces, and continuous $X$ and $Y$ with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and/or continuous $X$ and $Y$, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed "variational IB" method on several real-world datasets.

References in corpus (9)

• Opening the Black Box of Deep Neural Networks via Information
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Nonlinear Information Bottleneck
• Estimating Mixture Entropy with Pairwise Distances
• Uncertainty in the Variational Information Bottleneck
• The Convex Information Bottleneck Lagrangian
• Caveats for information bottleneck in deterministic scenarios
• Information Dropout: Learning Optimal Representations Through Noisy Computation

Cited by in corpus (19)

• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Nonlinear Information Bottleneck
• Entropy and mutual information in models of deep neural networks
• MSTREAM: Fast Anomaly Detection in Multi-Aspect Streams
• On Information Plane Analyses of Neural Network Classifiers -- A Review
• InfoBot: Transfer and Exploration via the Information Bottleneck
• A robust estimator of mutual information for deep learning interpretability
• Learning Optimal Representations with the Decodable Information Bottleneck
• Caveats for information bottleneck in deterministic scenarios
• PAC-Bayes Information Bottleneck
• Information Bottleneck and its Applications in Deep Learning
• Information Bottleneck Theory on Convolutional Neural Networks
• On the Effect of Low-Rank Weights on Adversarial Robustness of Neural Networks
• Information-Theoretic Abstractions for Resource-Constrained Agents via Mixed-Integer Linear Programming
• Gaussian Lower Bound for the Information Bottleneck Limit
• Computationally Efficient Approximations for Matrix-based Renyi's Entropy
• Do Compressed Representations Generalize Better?
• A Provably Convergent Information Bottleneck Solution via ADMM
• Gaussian Information Bottleneck and the Non-Perturbative Renormalization Group