Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sources

Yipei Xu, Dakuan Lu, Jiaqing Liang +7

Pre-trained language models (PLMs) have established the new paradigm in the field of NLP. For more powerful PLMs, one of the most popular and successful way is to continuously scal…