Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention

Luis Zuin, Alexis Huet, Dario Rossi +1

Self-distillation with privileged context adapts a language model from demonstrations by letting the model, once conditioned on a reference response, teach its context-free copy to…