Comparing Sequential Forecasters
arXiv:2110.00115 · doi:10.1287/opre.2021.0792
Abstract
Consider two forecasters, each making a single prediction for a sequence of events over time. We ask a relatively basic question: how might we compare these forecasters, either online or post-hoc, while avoiding unverifiable assumptions on how the forecasts and outcomes were generated? In this paper, we present a rigorous answer to this question by designing novel sequential inference procedures for estimating the time-varying difference in forecast scores. To do this, we employ confidence sequences (CS), which are sequences of confidence intervals that can be continuously monitored and are valid at arbitrary data-dependent stopping times ("anytime-valid"). The widths of our CSs are adaptive to the underlying variance of the score differences. Underlying their construction is a game-theoretic statistical framework, in which we further identify e-processes and p-processes for sequentially testing a weak null hypothesis -- whether one forecaster outperforms another on average (rather than always). Our methods do not make distributional assumptions on the forecasts or outcomes; our main theorems apply to any bounded scores, and we later provide alternative methods for unbounded scores. We empirically validate our approaches by comparing real-world baseball and weather forecasters.
arXiv commentPublished in Operations Research. Code and data sources available at https://github.com/yjchoe/ComparingForecasters
References in corpus (14)
- Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
- Statistical Postprocessing for Weather Forecasts -- Review, Challenges and Avenues in a Big Data World
- lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits
- E-values: Calibration, combination, and applications
- Time-uniform, nonparametric, nonasymptotic confidence sequences
- Theory and Applications of Proper Scoring Rules
- Time-uniform Chernoff bounds via nonnegative supermartingales
- Test Martingales, Bayes Factors and -Values
- Always Valid Inference: Bringing Sequential Analysis to A/B Testing
- Isotonic Distributional Regression
- Sequential estimation of quantiles with applications to A/B-testing and best-arm identification
- Admissible anytime-valid sequential inference must rely on nonnegative martingales
- Evaluating probability forecasts
- A Bandit Approach to Multiple Testing with False Discovery Control