AdaSwitch: Balancing Exploration and Guidance in Knowledge Distillation via Adaptive Switching

Jingyu Peng, Maolin Wang, Hengyi Cai +5

Small language models (SLMs) are crucial for applications with strict latency and computational constraints, yet achieving high performance remains challenging. Knowledge distillat…