Predicting Classification Accuracy When Adding New Unobserved Classes

Yuli Slavutsky, Yuval Benjamini

Multiclass classifiers are often designed and evaluated only on a sample from the classes on which they will eventually be applied. Hence, their final accuracy remains unknown. In…