1 paper
Zhenhe Wu, Jian Yang, Yaping Jin +3
Knowledge distillation (KD) improves language-model pretraining by matching teacher distributions at corpus prefixes, but provides no direct supervision at contexts induced by stud…