KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Tasks

Rishabh Bhardwaj, Tushar Vaidya, Soujanya Poria

We propose a new approach, Knowledge Distillation using Optimal Transport (KNOT), to distill the natural language semantic knowledge from multiple teacher networks to a student net…