Optimizing Teacher-Student Partitioning for Scalable Knowledge Distillation on HPC Systems

Adrian P. Dieguez, Victor Conchello Vendrell, Alex Batlle +3

Knowledge Distillation (KD) enables training smaller student models under the guidance of larger teacher models, and the widely adopted TRL library implements it. Yet, TRL treats b…