Information-Theoretic Understanding of Population Risk Improvement with Model Compression

Yuheng Bu, Weihao Gao, Shaofeng Zou +1

We show that model compression can improve the population risk of a pre-trained model, by studying the tradeoff between the decrease in the generalization error and the increase in…