Information Bottleneck and its Applications in Deep Learning

Abstract

Information Theory (IT) has been used in Machine Learning (ML) from early days of this field. In the last decade, advances in Deep Neural Networks (DNNs) have led to surprising improvements in many applications of ML. The result has been a paradigm shift in the community toward revisiting previous ideas and applications in this new framework. Ideas from IT are no exception. One of the ideas which is being revisited by many researchers in this new era, is Information Bottleneck (IB); a formulation of information extraction based on IT. The IB is promising in both analyzing and improving DNNs. The goal of this survey is to review the IB concept and demonstrate its applications in deep learning. The information theoretic nature of IB, makes it also a good candidate in showing the more general concept of how IT can be used in ML. Two important concepts are highlighted in this narrative on the subject, i) the concise and universal view that IT provides on seemingly unrelated methods of ML, demonstrated by explaining how IB relates to minimal sufficient statistics, stochastic gradient descent, and variational auto-encoders, and ii) the common technical mistakes and problems caused by applying ideas from IT, which is discussed by a careful study of some recent methods suffering from them.

References in corpus (18)

• Opening the Black Box of Deep Neural Networks via Information
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• InfoVAE: Information Maximizing Variational Autoencoders
• Train faster, generalize better: Stability of stochastic gradient descent
• Generalized Denoising Auto-Encoders as Generative Models
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Nonlinear Information Bottleneck
• Multivariate Information Bottleneck
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Mathematics of Deep Learning
• Denoising Criterion for Variational Auto-Encoding Framework
• Learners that Use Little Information
• Fixing a Broken ELBO
• How much does your data exploration overfit? Controlling bias via information usage
• Degeneration in VAE: in the Light of Fisher Information Loss
• Flow of Information in Feed-Forward Deep Neural Networks
• The Role of Information Complexity and Randomization in Representation Learning
• A Direct Sum Result for the Information Complexity of Learning

Cited by in corpus (4)

• On Information Plane Analyses of Neural Network Classifiers -- A Review
• How higher goals are constructed and collapse under stress: a hierarchical Bayesian control systems perspective
• Variational Encoders and Autoencoders : Information-theoretic Inference and Closed-form Solutions
• Think Global, Act Local: Relating DNN generalisation and node-level SNR