Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learning

Abstract

The overall predictive uncertainty of a trained predictor can be decomposed into separate contributions due to epistemic and aleatoric uncertainty. Under a Bayesian formulation, assuming a well-specified model, the two contributions can be exactly expressed (for the log-loss) or bounded (for more general losses) in terms of information-theoretic quantities (Xu and Raginsky, 2020). This paper addresses the study of epistemic uncertainty within an information-theoretic framework in the broader setting of Bayesian meta-learning. A general hierarchical Bayesian model is assumed in which hyperparameters determine the per-task priors of the model parameters. Exact characterizations (for the log-loss) and bounds (for more general losses) are derived for the epistemic uncertainty -quantified by the minimum excess meta-risk (MEMR)- of optimal meta-learning rules. This characterization is leveraged to bring insights into the dependence of the epistemic uncertainty on the number of tasks and on the amount of per-task training data. Experiments are presented that use the proposed information-theoretic bounds, evaluated via neural mutual information estimators, to compare the performance of conventional learning and meta-learning as the number of meta-learning tasks increases.

arXiv commentUnder review

References in corpus (12)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Matching Networks for One Shot Learning
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Bayesian Model-Agnostic Meta-Learning
• A PAC-Bayesian bound for Lifelong Learning
• Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• The Case for Bayesian Deep Learning
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
• Probabilistic Model-Agnostic Meta-Learning
• Reasoning About Generalization via Conditional Mutual Information