SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines

Itai Morad, Nir Shlezinger, Yonina C. Eldar

Knowledge Distillation (KD) is a central paradigm for transferring knowledge from a large teacher network to a typically smaller student model, often by leveraging soft probabilist…