Taking Notes on the Fly Helps BERT Pre-training

Qiyu Wu, Chen Xing, Yatao Li +3

How to make unsupervised language pre-training more efficient and less resource-intensive is an important research direction in NLP. In this paper, we focus on improving the effici…