The Conditional Entropy Bottleneck
arXiv:2002.05379 · doi:10.3390/e22090999
Abstract
Much of the field of Machine Learning exhibits a prominent set of failure modes, including vulnerability to adversarial examples, poor out-of-distribution (OoD) detection, miscalibration, and willingness to memorize random labelings of datasets. We characterize these as failures of robust generalization, which extends the traditional measure of generalization as accuracy or related metrics on a held-out set. We hypothesize that these failures to robustly generalize are due to the learning systems retaining too much information about the training data. To test this hypothesis, we propose the Minimum Necessary Information (MNI) criterion for evaluating the quality of a model. In order to train models that perform well with respect to the MNI criterion, we present a new objective function, the Conditional Entropy Bottleneck (CEB), which is closely related to the Information Bottleneck (IB). We experimentally test our hypothesis by comparing the performance of CEB models with deterministic models and Variational Information Bottleneck (VIB) models on a variety of different datasets and robustness challenges. We find strong empirical evidence supporting our hypothesis that MNI models improve on these problems of robust generalization.
References in corpus (30)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Auto-Encoding Variational Bayes
- Representation Learning with Contrastive Predictive Coding
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- WaveNet: A Generative Model for Raw Audio
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- The information bottleneck method
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Learning deep representations by mutual information estimation and maximization
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Towards Deep Learning Models Resistant to Adversarial Attacks
- Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
- Understanding deep learning requires rethinking generalization
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- AutoAugment: Learning Augmentation Policies from Data
- Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
- Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
- Adversarial Examples Are Not Bugs, They Are Features
- Learning Confidence for Out-of-Distribution Detection in Neural Networks
- Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
- Implicit Reparameterization Gradients
- Do CIFAR-10 Classifiers Generalize to CIFAR-10?
- Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
- SCAN: Learning Hierarchical Compositional Visual Concepts
- Uncertainty in the Variational Information Bottleneck
- Do Deep Generative Models Know What They Don't Know?
- Generative Models of Visually Grounded Imagination
- Learners that Use Little Information
- Learnability for the Information Bottleneck
Cited by in corpus (12)
- What Makes for Good Views for Contrastive Learning?
- Generalization in Quantum Machine Learning: a Quantum Information Perspective
- Graph Information Bottleneck
- Action and Perception as Divergence Minimization
- Predictive Information Accelerates Learning in RL
- An Unsupervised Information-Theoretic Perceptual Quality Metric
- CEB Improves Model Robustness
- Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
- DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
- Pareto-optimal clustering with the primal deterministic information bottleneck
- Entropy Minimization In Emergent Languages
- Compressive Visual Representations