Weighted Sampling for Masked Language Modeling

Linhan Zhang, Qian Chen, Wen Wang +5

Masked Language Modeling (MLM) is widely used to pretrain language models. The standard random masking strategy in MLM causes the pre-trained language models (PLMs) to be biased to…