Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle

Abstract

In this theory paper, we investigate training deep neural networks (DNNs) for classification via minimizing the information bottleneck (IB) functional. We show that the resulting optimization problem suffers from two severe issues: First, for deterministic DNNs, either the IB functional is infinite for almost all values of network parameters, making the optimization problem ill-posed, or it is piecewise constant, hence not admitting gradient-based optimization methods. Second, the invariance of the IB functional under bijections prevents it from capturing properties of the learned representation that are desirable for classification, such as robustness and simplicity. We argue that these issues are partly resolved for stochastic DNNs, DNNs that include a (hard or soft) decision rule, or by replacing the IB functional with related, but more well-behaved cost functions. We conclude that recent successes reported about training DNNs using the IB framework must be attributed to such solutions. As a side effect, our results indicate limitations of the IB framework for the analysis of DNNs. We also note that rather than trying to repair the inherent problems in the IB functional, a better approach may be to design regularizers on latent representation enforcing the desired properties directly.

16 pages, to appear in IEEE Trans. Pattern Analysis and Machine Intelligence

References in corpus (14)

• The information bottleneck method
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Opening the Black Box of Deep Neural Networks via Information
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• A Closer Look at Memorization in Deep Networks
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Nonlinear Information Bottleneck
• Invariant Representations without Adversarial Training
• Dataset Augmentation in Feature Space
• Uncertainty in the Variational Information Bottleneck
• Forward Thinking: Building and Training Neural Networks One Layer at a Time
• Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation
• Ensemble Robustness and Generalization of Stochastic Deep Learning Algorithms
• Neural Networks Regularization Through Class-wise Invariant Representation Learning

Cited by in corpus (47)

• Nonlinear Information Bottleneck
• Why should we add early exits to neural networks?
• The Conditional Entropy Bottleneck
• On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views
• Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations
• Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustness
• On Information Plane Analyses of Neural Network Classifiers -- A Review
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
• Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation
• Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels
• The Convex Information Bottleneck Lagrangian
• Estimating Information Flow in Deep Neural Networks
• Learning Optimal Representations with the Decodable Information Bottleneck
• Caveats for information bottleneck in deterministic scenarios
• InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle
• Discovering and Explaining the Representation Bottleneck of DNNs
• Information Bottleneck and its Applications in Deep Learning
• Learning to Learn with Variational Information Bottleneck for Domain Generalization
• On the Information Plane of Autoencoders
• Bottleneck Problems: Information and Estimation-Theoretic View
• Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
• Pareto-optimal data compression for binary classification tasks
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• Information in Infinite Ensembles of Infinitely-Wide Neural Networks
• Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness
• Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration
• Renormalized Mutual Information for Artificial Scientific Discovery
• Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional
• Measuring Dependence with Matrix-based Entropy Functional
• Information Bottleneck Theory on Convolutional Neural Networks
• TURBO: The Swiss Knife of Auto-Encoders
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training
• Information Bottleneck: Exact Analysis of (Quantized) Neural Networks
• Information Plane Analysis Visualization in Deep Learning via Transfer Entropy
• Do Compressed Representations Generalize Better?
• Optimization Induced Equilibrium Networks
• Feature selection of neural networks is skewed towards the less abstract cue
• Ranking Feature-Block Importance in Artificial Multiblock Neural Networks
• Understanding Neural Networks with Logarithm Determinant Entropy Estimator
• BiDet: An Efficient Binarized Object Detector
• A Class of Nonbinary Symmetric Information Bottleneck Problems
• InfoNEAT: Information Theory-based NeuroEvolution of Augmenting Topologies for Side-channel Analysis
• An Information Bottleneck Problem with Rényi's Entropy
• Class-Conditional Compression and Disentanglement: Bridging the Gap between Neural Networks and Naive Bayes Classifiers
• Understanding the Behaviour of the Empirical Cross-Entropy Beyond the Training Distribution
• Disentangled Variational Information Bottleneck for Multiview Representation Learning