Rate-Distortion Analysis of Minimum Excess Risk in Bayesian Learning

Abstract

In parametric Bayesian learning, a prior is assumed on the parameter $W$ which determines the distribution of samples. In this setting, Minimum Excess Risk (MER) is defined as the difference between the minimum expected loss achievable when learning from data and the minimum expected loss that could be achieved if $W$ was observed. In this paper, we build upon and extend the recent results of (Xu & Raginsky, 2020) to analyze the MER in Bayesian learning and derive information-theoretic bounds on it. We formulate the problem as a (constrained) rate-distortion optimization and show how the solution can be bounded above and below by two other rate-distortion functions that are easier to study. The lower bound represents the minimum possible excess risk achievable by any process using $R$ bits of information from the parameter $W$. For the upper bound, the optimization is further constrained to use $R$ bits from the training set, a setting which relates MER to information-theoretic bounds on the generalization gap in frequentist learning. We derive information-theoretic bounds on the difference between these upper and lower bounds and show that they can provide order-wise tight rates for MER under certain conditions. This analysis gives more insight into the information-theoretic nature of Bayesian learning as well as providing novel bounds.

arXiv commentAccepted at ICML 2021

References in corpus (8)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Information-theoretic analysis of generalization capability of learning algorithms
• Learners that Use Little Information
• Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms
• How much does your data exploration overfit? Controlling bias via information usage
• The Shannon Lower Bound is Asymptotically Tight
• Rate Distortion For Model Compression: From Theory To Practice
• Reasoning About Generalization via Conditional Mutual Information

Cited by in corpus (1)

• An Information-Theoretic Analysis of The Cost of Decentralization for Learning and Inference Under Privacy Constraints