most citedAlexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems

64 citations · 64 across the 1 of their papers we have counts for

collaborators

1 paper

cs.CL2022★ 64 cited

Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems

Jack FitzGerald, Shankar Ananthakrishnan, Konstantine Arkoudas +38

We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B, their subsequent distillation into smaller m…