Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models

Yao Fu, Yin Yu, Xiaotian Han +4

Knowledge distillation (KD) has become a widely adopted approach for compressing large language models (LLMs) to reduce computational costs and memory footprints. However, the avai…