The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering

Adam Noonan

Many machine-learning systems set a threshold at a quantile of a calibration set: conformal predictors that promise 90% coverage by drawing their cutoff at the calibration set's 90…