Efficient One Pass Self-distillation with Zipf's Label Smoothing

Jiajun Liang, Linze Li, Zhaodong Bing +4

Self-distillation exploits non-uniform soft supervision from itself during training and improves performance without any runtime cost. However, the overhead during training is ofte…