Causal Distillation for Language Models

Zhengxuan Wu, Atticus Geiger, Josh Rozner +5

Distillation efforts have led to language models that are more compact and efficient without serious drops in performance. The standard approach to distillation trains a student mo…