Distilling Large Language Models into Tiny and Effective Students using pQRNN

Prabhu Kaliamoorthi, Aditya Siddhant, Edward Li +1

Large pre-trained multilingual models like mBERT, XLM-R achieve state of the art results on language understanding tasks. However, they are not well suited for latency critical app…