Large scale distributed neural network training through online distillation

Rohan Anil, Gabriel Pereyra, Alexandre Passos +3

Techniques such as ensembling and distillation promise model quality improvements when paired with almost any base model. However, due to increased test-time cost (for ensembles) a…