Variance-reduced Language Pretraining via a Mask Proposal Network

Liang Chen

Self-supervised learning, a.k.a., pretraining, is important in natural language processing. Most of the pretraining methods first randomly mask some positions in a sentence and the…