On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models

Sean Farhat, Deming Chen

In this paper, we propose that small models may not need to absorb the cost of pre-training to reap its benefits. Instead, they can capitalize on the astonishing results achieved b…