What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation

Kanghui Tian, Siyuan Liu, Tianxiang Jiang +8

More privileged information does not always make a better teacher. We study this tension in on-policy self-distillation (OPSD), where a frozen copy of the base model scores the stu…