Moment Multicalibration for Uncertainty Estimation

Abstract

We show how to achieve the notion of "multicalibration" from Hébert-Johnson et al. [2018] not just for means, but also for variances and other higher moments. Informally, it means that we can find regression functions which, given a data point, can make point predictions not just for the expectation of its label, but for higher moments of its label distribution as well-and those predictions match the true distribution quantities when averaged not just over the population as a whole, but also when averaged over an enormous number of finely defined subgroups. It yields a principled way to estimate the uncertainty of predictions on many different subgroups-and to diagnose potential sources of unfairness in the predictive power of features across subgroups. As an application, we show that our moment estimates can be used to derive marginal prediction intervals that are simultaneously valid as averaged over all of the (sufficiently large) subgroups for which moment multicalibration has been obtained.

References in corpus (4)

• A tutorial on conformal prediction
• Fairness Through Computationally-Bounded Awareness
• Probably Approximately Metric-Fair Learning
• Sample Complexity of Uniform Convergence for Multicalibration

Cited by in corpus (10)

• Assessing Generalization of SGD via Disagreement
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Private Prediction Sets
• Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration
• Online Multivalid Learning: Means, Moments, and Prediction Intervals
• Stronger Calibration Lower Bounds via Sidestepping
• Understanding the Under-Coverage Bias in Uncertainty Estimation
• Multi-group Agnostic PAC Learnability
• Explaining generalization in deep learning: progress and fundamental limits
• Outcome Indistinguishability