A Theory of Usable Information Under Computational Constraints

Abstract

We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer. The resulting \emph{predictive $\mathcal{V}$-information} encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon's mutual information and in violation of the data processing inequality, $\mathcal{V}$-information can be created through computation. This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning. Additionally, we show that by incorporating computational constraints, $\mathcal{V}$-information can be reliably estimated from data even in high dimensions with PAC-style guarantees. Empirically, we demonstrate predictive $\mathcal{V}$-information is more effective than mutual information for structure learning and fair representation learning.

arXiv commentICLR 2020 (Talk)

References in corpus (6)

• Auto-Encoding Variational Bayes
• Estimating Mutual Information
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• Information Theoretical Estimators Toolbox
• Learning Controllable Fair Representations

Cited by in corpus (15)

• On Mutual Information Maximization for Representation Learning
• Acquisition of Chess Knowledge in AlphaZero
• Learning Optimal Representations with the Decodable Information Bottleneck
• Layer-wise Characterization of Latent Information Leakage in Federated Learning
• Evaluating representations by the complexity of learning low-loss predictors
• Sliced Mutual Information: A Scalable Measure of Statistical Dependence
• Usable Information and Evolution of Optimal Representations During Training
• Training Normalizing Flows with the Information Bottleneck for Competitive Generative Classification
• What Context Features Can Transformer Language Models Use?
• Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variations
• TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learning
• Predicting the Accuracy of a Few-Shot Classifier
• Fair Normalizing Flows
• A Practical & Unified Notation for Information-Theoretic Quantities in ML
• Conditional probing: measuring usable information beyond a baseline