Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting

Mingqi Wu, Archer Y. Yang, Qiang Sun

Iterative self-training (self-distillation) repeatedly refits a model on pseudo-labels generated by its own predictions. We study this procedure in overparameterized linear regress…