A Compact Pretraining Approach for Neural Language Models

Shahriar Golchin, Mihai Surdeanu, Nazgol Tavabi +1

Domain adaptation for large neural language models (NLMs) is coupled with massive amounts of unstructured data in the pretraining phase. In this study, however, we show that pretra…