Calibration scoring under dependent seed variables in Cooke's Classical Model: a Mehler-Hermite design-effect correction

Abstract

Cooke's Classical Model scores the statistical accuracy of experts by the $p$-value of a likelihood ratio test that compares the bin frequencies of realized seed variables with nominal probabilities, using a ${\chi }^2$ reference. This calibration score drives performance-based weights and implicitly assumes independent seed variables, although seeds often share a common driver, such as the same indicator in several regions or periods. Under dependence the statistic converges to a weighted sum of ${\chi }_1^2$ variables, and positive dependence makes the classical score anti-conservative. Under a Gaussian-copula working model, a Mehler-Hermite expansion gives the long-run covariance of the bin indicators in closed form, defines a design effect and an effective number of seed variables, and shows that the first-order correction coincides with the ad hoc calibration power of existing software. Rao-Scott-type and parametric-bootstrap corrected scores are proposed, together with a restricted maximum likelihood estimator of the nuisance dependence, computed from rank normal scores and invariant to location-scale miscalibration, and a prepivoted bootstrap that propagates its uncertainty. In simulations the classical score rejects up to 68\% of calibrated experts at a nominal 5\% level, whereas the corrected scores remove most of the excess rejection. In 45 published studies, design effects of 1.25-2 shift more than 5\% of the optimized weight in 29-44\% of studies. In three social-science studies with structured seeds, the baseline estimates imply that 18-21 seed variables carry the information of only 11-13 independent ones, although conclusions about individual experts depend on how the dependence is specified.