Adversarial Moment-Matching Distillation of Large Language Models

Chen Jia

Knowledge distillation (KD) has been shown to be highly effective in guiding a student model with a larger teacher model and achieving practical benefits in improving the computati…