Distilling BERT into Simple Neural Networks with Unlabeled Transfer Data

Subhabrata Mukherjee, Ahmed Hassan Awadallah

Recent advances in pre-training huge models on large amounts of text through self supervision have obtained state-of-the-art results in various natural language processing tasks. H…