When Do Language Models Add Value? Relative Competence in Hybrid Event Forecasting

Abstract

When a forecasting system already has a prediction from a market, crowd, or statistical model, when should it use a language model? Standalone accuracy cannot answer this question because a less accurate model may still improve the existing forecast. We study this distinction through relative competence, separating a model's replacement advantage from its complementary value. Under Brier loss (squared error between predicted probabilities and binary outcomes), we characterize when model disagreement can correct external forecast errors and derive the population gain from regime-specific affine pooling. We introduce a competence gate that estimates source weights from resolved outcomes, shrinks regime-specific estimates toward a global solution, and chronologically recalibrates the pooled forecast. Across 4,974 resolved binary questions and four language models, the best standalone model has 24.94% higher Brier loss than the calibrated external forecast, yet the gate improves that reference by 3.35% (95% CI, 1.80 to 4.90%). Model contributions vary across forecasting regimes, with larger corrections in thinner markets and forecasting crowds and smaller corrections in higher-participation settings. The gate also shows no statistically detectable performance difference from more flexible contextual aggregators. These results demonstrate that standalone superiority is not necessary for complementary predictive value and that language models should be evaluated relative to the forecasts already available.

arXiv comment18 pages, including references and appendices