On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views

Abstract

This tutorial paper focuses on the variants of the bottleneck problem taking an information theoretic perspective and discusses practical methods to solve it, as well as its connection to coding and learning aspects. The intimate connections of this setting to remote source-coding under logarithmic loss distortion measure, information combining, common reconstruction, the Wyner-Ahlswede-Korner problem, the efficiency of investment information, as well as, generalization, variational inference, representation learning, autoencoders, and others are highlighted. We discuss its extension to the distributed information bottleneck problem with emphasis on the Gaussian model and highlight the basic connections to the uplink Cloud Radio Access Networks (CRAN) with oblivious processing. For this model, the optimal trade-offs between relevance (i.e., information) and complexity (i.e., rates) in the discrete and vector Gaussian frameworks is determined. In the concluding outlook, some interesting problems are mentioned such as the characterization of the optimal inputs ("features") distributions under power limitations maximizing the "relevance" for the Gaussian information bottleneck, under "complexity" constraints.

To be published in Entropy as part of the Special Issue Information Theory for Data Communications and Processing. 51 pages. arXiv admin note: text overlap with arXiv:1807.04193

References in corpus (9)

• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Information-theoretic analysis of generalization capability of learning algorithms
• Estimation Efficiency Under Privacy Constraints
• Analog-to-Digital Compression: A New Paradigm for Converting Signals to Bits
• Justification of Logarithmic Loss via the Benefit of Side Information
• Channel Capacity under Sub-Nyquist Nonuniform Sampling
• The Convex Information Bottleneck Lagrangian
• On the Difference Between the Information Bottleneck and the Deep Information Bottleneck
• Decoding of Non-Binary LDPC Codes Using the Information Bottleneck Method

Cited by in corpus (8)

• Semantic Information Recovery in Wireless Networks
• Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding
• The Broadcast Approach in Communication Networks
• Least $k$th-Order and Rényi Generative Adversarial Networks
• Information flows of diverse autoencoders
• Bottleneck Problems: Information and Estimation-Theoretic View
• Theoretical Analysis of Deep Neural Networks in Physical Layer Communication
• Differential Entropy of the Conditional Expectation under Additive Gaussian Noise