Knowledge Distillation for Efficient Sequences of Training Runs

Xingyu Liu, Alex Leonardi, Lu Yu +3

In many practical scenarios -- like hyperparameter search or continual retraining with new data -- related training runs are performed many times in sequence. Current practice is t…