The Distributional View of Knowledge Distillation

Gordei Verbii, Juho Lee

Token-level knowledge distillation (KD) matches two conditional distributions per position, yet the standard objectives compare them pointwise: a Kullback-Leibler gradient is blind…