Embedding Compression for Teacher-to-Student Knowledge Transfer

Yiwei Ding, Alexander Lerch

Common knowledge distillation methods require the teacher model and the student model to be trained on the same task. However, the usage of embeddings as teachers has also been pro…