The Conditional Entropy Bottleneck

Abstract

Much of the field of Machine Learning exhibits a prominent set of failure modes, including vulnerability to adversarial examples, poor out-of-distribution (OoD) detection, miscalibration, and willingness to memorize random labelings of datasets. We characterize these as failures of robust generalization, which extends the traditional measure of generalization as accuracy or related metrics on a held-out set. We hypothesize that these failures to robustly generalize are due to the learning systems retaining too much information about the training data. To test this hypothesis, we propose the Minimum Necessary Information (MNI) criterion for evaluating the quality of a model. In order to train models that perform well with respect to the MNI criterion, we present a new objective function, the Conditional Entropy Bottleneck (CEB), which is closely related to the Information Bottleneck (IB). We experimentally test our hypothesis by comparing the performance of CEB models with deterministic models and Variational Information Bottleneck (VIB) models on a variety of different datasets and robustness challenges. We find strong empirical evidence supporting our hypothesis that MNI models improve on these problems of robust generalization.

References in corpus (4)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• WaveNet: A Generative Model for Raw Audio
• Understanding deep learning requires rethinking generalization

Cited by in corpus (11)

• What Makes for Good Views for Contrastive Learning?
• Generalization in Quantum Machine Learning: a Quantum Information Perspective
• Graph Information Bottleneck
• Predictive Information Accelerates Learning in RL
• Action and Perception as Divergence Minimization
• An Unsupervised Information-Theoretic Perceptual Quality Metric
• CEB Improves Model Robustness
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• Pareto-optimal clustering with the primal deterministic information bottleneck
• Compressive Visual Representations