The advantages of multiple classes for reducing overfitting from test set reuse

Vitaly Feldman, Roy Frostig, Moritz Hardt

Excessive reuse of holdout data can lead to overfitting. However, there is little concrete evidence of significant overfitting due to holdout reuse in popular multiclass benchmarks…