Knowledge Distillation via the Target-aware Transformer

Sihao Lin, Hongwei Xie, Bing Wang +4

Knowledge distillation becomes a de facto standard to improve the performance of small neural networks. Most of the previous works propose to regress the representational features…