Towards Optimal Trade-offs in Knowledge Distillation for CNNs and Vision Transformers at the Edge

John Violos, Symeon Papadopoulos, Ioannis Kompatsiaris

This paper discusses four facets of the Knowledge Distillation (KD) process for Convolutional Neural Networks (CNNs) and Vision Transformer (ViT) architectures, particularly when e…