Deep Variational Information Bottleneck

Abstract

We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.

19 pages, 8 figures, Accepted to ICLR17

References in corpus (8)

• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Learning with a Strong Adversary
• Differential Privacy as a Mutual Information Constraint
• Robustness of classifiers: from adversarial to random noise
• Deep Variational Canonical Correlation Analysis
• Universal adversarial perturbations
• Relevant sparse codes with variational information bottleneck
• Confusing Deep Convolution Networks by Relabelling

Cited by in corpus (160)

• What Makes for Good Views for Contrastive Learning?
• Recent Advances in Autoencoder-Based Representation Learning
• DisenHAN: Disentangled Heterogeneous Graph Attention Network for Recommendation
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Isolating Sources of Disentanglement in Variational Autoencoders
• Dream to Control: Learning Behaviors by Latent Imagination
• Compressing Neural Networks using the Variational Information Bottleneck
• Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder
• Invariant Representations without Adversarial Training
• Deep matrix factorizations
• Deep Variational Canonical Correlation Analysis
• Graph Information Bottleneck
• A survey on intrinsic motivation in reinforcement learning
• Mathematics of Deep Learning
• Revisiting Training Strategies and Generalization Performance in Deep Metric Learning
• Meta-Learning without Memorization
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Machine Theory of Mind
• Variational Autoencoders for Collaborative Filtering
• Uncertainty in the Variational Information Bottleneck
• Towards Evaluating the Robustness of Deep Diagnostic Models by Adversarial Attack
• Excessive Invariance Causes Adversarial Vulnerability
• Meta reinforcement learning as task inference
• Mapping Machine-Learned Physics into a Human-Readable Space
• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• Where is the Information in a Deep Neural Network?
• Robust Training of Vector Quantized Bottleneck Models
• Fixing a Broken ELBO
• Learning Efficient Multi-agent Communication: An Information Bottleneck Approach
• Understanding the Limitations of Variational Mutual Information Estimators
• The Functional Neural Process
• Information Theoretic Counterfactual Learning from Missing-Not-At-Random Feedback
• Improving Zero-shot Voice Style Transfer via Disentangled Representation Learning
• AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation
• Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
• Significance-aware Information Bottleneck for Domain Adaptive Semantic Segmentation
• Information bottleneck through variational glasses
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• The Deep Kernelized Autoencoder
• Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments
• TherML: Thermodynamics of Machine Learning
• Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Caveats for information bottleneck in deterministic scenarios
• Dynamics-aware Embeddings
• Phase Transitions for the Information Bottleneck in Representation Learning
• Behavior Priors for Efficient Reinforcement Learning
• Generating Tertiary Protein Structures via an Interpretative Variational Autoencoder
• Learning the Redundancy-free Features for Generalized Zero-Shot Object Recognition
• Explaining a black-box using Deep Variational Information Bottleneck Approach
• Unified Adversarial Invariance
• Theory and Evaluation Metrics for Learning Disentangled Representations
• Training Invertible Neural Networks as Autoencoders
• The Dual Information Bottleneck
• Diversity inducing Information Bottleneck in Model Ensembles
• An Information Theory-inspired Strategy for Automatic Network Pruning
• Multi-task Batch Reinforcement Learning with Metric Learning
• Improving Unsupervised Domain Adaptation with Variational Information Bottleneck
• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
• Variational Predictive Information Bottleneck
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• On the Maximum Mutual Information Capacity of Neural Architectures
• PointMask: Towards Interpretable and Bias-Resilient Point Cloud Processing
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• Information Obfuscation of Graph Neural Networks
• On the Fairness of Disentangled Representations
• Disentangled Representations from Non-Disentangled Models
• flexgrid2vec: Learning Efficient Visual Representations Vectors
• Contrastive Variational Reinforcement Learning for Complex Observations
• Information Potential Auto-Encoders
• Neuron Campaign for Initialization Guided by Information Bottleneck Theory
• Variational Autoencoder Analysis of Ising Model Statistical Distributions and Phase Transitions
• Recognizing Predictive Substructures with Subgraph Information Bottleneck
• Embedding Expansion: Augmentation in Embedding Space for Deep Metric Learning
• SOAC: The Soft Option Actor-Critic Architecture
• Discovery and Separation of Features for Invariant Representation Learning
• The Role of Information Complexity and Randomization in Representation Learning
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Learning Discrete State Abstractions With Deep Variational Inference
• Controllable Paraphrase Generation with a Syntactic Exemplar
• Information-Theoretic Abstractions for Resource-Constrained Agents via Mixed-Integer Linear Programming
• Improving Robustness to Model Inversion Attacks via Mutual Information Regularization
• Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers
• Measuring the Biases and Effectiveness of Content-Style Disentanglement
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• Information Bottleneck Constrained Latent Bidirectional Embedding for Zero-Shot Learning
• Independence Promoted Graph Disentangled Networks
• Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation
• On Predictive Information in RNNs
• A Batch Normalized Inference Network Keeps the KL Vanishing Away
• MetaInfoNet: Learning Task-Guided Information for Sample Reweighting
• S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learning
• Entropy Minimization In Emergent Languages
• Variational Information Bottleneck on Vector Quantized Autoencoders
• LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates
• Mutual Information State Intrinsic Control
• A Modular Deep Learning Pipeline for Galaxy-Scale Strong Gravitational Lens Detection and Modeling
• A Scalable Gradient-Free Method for Bayesian Experimental Design with Implicit Models
• Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
• Large scale evaluation of importance maps in automatic speech recognition
• Predicting with High Correlation Features
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Differentially Private Variational Autoencoders with Term-wise Gradient Aggregation
• Cross-domain Imitation from Observations
• Instance-Aware Graph Convolutional Network for Multi-Label Classification
• Optimization Induced Equilibrium Networks
• Robust Representation Learning via Perceptual Similarity Metrics
• Information Theoretic Interpretation of Deep learning
• Variational Information Bottleneck for Effective Low-resource Audio Classification
• A Free-Energy Principle for Representation Learning
• Likelihood Ratio Exponential Families
• Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval
• Free Energy Minimization: A Unified Framework for Modelling, Inference, Learning,and Optimization
• Information-Theoretic Odometry Learning
• Drill the Cork of Information Bottleneck by Inputting the Most Important Data
• Revisiting Factorizing Aggregated Posterior in Learning Disentangled Representations
• Stochastic-Shield: A Probabilistic Approach Towards Training-Free Adversarial Defense in Quantized CNNs
• Information Theory-Guided Heuristic Progressive Multi-View Coding
• Dynamic Narrowing of VAE Bottlenecks Using GECO and L0 Regularization
• Projection-wise Disentangling for Fair and Interpretable Representation Learning: Application to 3D Facial Shape Analysis
• Signature-Graph Networks
• Fighting Copycat Agents in Behavioral Cloning from Observation Histories
• The distance between the weights of the neural network is meaningful
• Kernelized Hashcode Representations for Relation Extraction
• Variational Knowledge Distillation for Disease Classification in Chest X-Rays
• Deep Latent-Variable Kernel Learning
• Predictive Uncertainty through Quantization
• Learning and Inference in Imaginary Noise Models
• Amanuensis: The Programmer's Apprentice
• Modeling Psychotherapy Dialogues with Kernelized Hashcode Representations: A Nonparametric Information-Theoretic Approach
• Margin Maximization as Lossless Maximal Compression
• PAC${}^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime
• EQ-Net: A Unified Deep Learning Framework for Log-Likelihood Ratio Estimation and Quantization
• Self-Paced Uncertainty Estimation for One-shot Person Re-Identification
• Information Bottleneck Approach to Spatial Attention Learning
• Explaining Representation by Mutual Information
• Variational Reward Estimator Bottleneck: Learning Robust Reward Estimator for Multi-Domain Task-Oriented Dialog
• Understanding the Behaviour of the Empirical Cross-Entropy Beyond the Training Distribution
• KL Guided Domain Adaptation
• Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments
• Adaptation of Quadruped Robot Locomotion with Meta-Learning
• Cause-Effect Deep Information Bottleneck For Systematically Missing Covariates
• Improving Generalization of Deep Networks for Inverse Reconstruction of Image Sequences
• Specializing Word Embeddings (for Parsing) by Information Bottleneck
• The Variational InfoMax Learning Objective
• Robust Disentanglement of a Few Factors at a Time
• Variational Information Bottleneck Model for Accurate Indoor Position Recognition
• An Information Bottleneck Problem with Rényi's Entropy
• Improve variational autoEncoder with auxiliary softmax multiclassifier
• Guess First to Enable Better Compression and Adversarial Robustness
• On Study of Mutual Information and its Estimation Methods
• Towards Better Understanding of Disentangled Representations via Mutual Information
• Quantization-Based Regularization for Autoencoders
• On Learning Prediction-Focused Mixtures
• A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations
• Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics