Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning

Abstract

The Information Bottleneck principle offers both a mechanism to explain how deep neural networks train and generalize, as well as a regularized objective with which to train models. However, multiple competing objectives are proposed in the literature, and the information-theoretic quantities used in these objectives are difficult to compute for large deep neural networks, which in turn limits their use as a training objective. In this work, we review these quantities and compare and unify previously proposed objectives, which allows us to develop surrogate objectives more friendly to optimization without relying on cumbersome tools such as density estimation. We find that these surrogate objectives allow us to apply the information bottleneck to modern neural network architectures. We demonstrate our insights on MNIST, CIFAR-10 and Imagenette with modern DNN architectures (ResNets).

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Representation Learning with Contrastive Predictive Coding
• Estimating Mutual Information
• The information bottleneck method
• Understanding deep learning requires rethinking generalization
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Opening the Black Box of Deep Neural Networks via Information
• Bayesian Active Learning for Classification and Preference Learning
• Adversarial Machine Learning at Scale
• Deep Variational Information Bottleneck
• BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
• On Mutual Information Maximization for Representation Learning
• Understanding disentangling in $\beta$-VAE
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• From Variational to Deterministic Autoencoders
• The Conditional Entropy Bottleneck
• Formal Limitations on the Measurement of Mutual Information
• Multi-Sample Dropout for Accelerated Training and Better Generalization
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors

Cited by in corpus (5)

• Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization
• Action and Perception as Divergence Minimization
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• A Closer Look at the Adversarial Robustness of Information Bottleneck Models
• A Practical & Unified Notation for Information-Theoretic Quantities in ML