Knowledge Distillation with Training Wheels

Guanlin Liu, Anand Ramachandran, Tanmay Gangwani +2

Knowledge distillation is used, in generative language modeling, to train a smaller student model using the help of a larger teacher model, resulting in improved capabilities for t…