Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty

Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implicitly assuming the teacher to…