Teacher Guided Training: An Efficient Framework for Knowledge Transfer

Manzil Zaheer, Ankit Singh Rawat, Seungyeon Kim +5

The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling…