VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale

Zhiwei Hao, Jianyuan Guo, Kai Han +3

The tremendous success of large models trained on extensive datasets demonstrates that scale is a key ingredient in achieving superior results. Therefore, the reflection on the rat…