2 papers
cs.CV2025
Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformers
Huiyuan Tian, Bonan Xu, Shijian Li
While feature-based knowledge distillation has proven highly effective for compressing CNNs, these techniques unexpectedly fail when applied to Vision Transformers (ViTs), often pe…
cs.CV2025
SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis
Huiyuan Tian, Bonan Xu, Shijian Li +1
Knowledge Distillation (KD) has achieved widespread success in compressing large Vision Transformers (ViTs), but a unified theoretical framework for both ViTs and KD is still lacki…