Linear Projections of Teacher Embeddings for Few-Class Distillation

Noel Loo, Fotis Iliopoulos, Wei Hu +1

Knowledge Distillation (KD) has emerged as a promising approach for transferring knowledge from a larger, more complex teacher model to a smaller student model. Traditionally, KD i…