Efficient Transformer Knowledge Distillation: A Performance Review

Nathan Brown, Ashton Williamson, Tahj Anderson +1

As pretrained transformer language models continue to achieve state-of-the-art performance, the Natural Language Processing community has pushed for advances in model compression a…