POSPAN: Position-Constrained Span Masking for Language Model Pre-training

Zhenyu Zhang, Lei Shen, Yuming Zhao +2

Span-level masked language modeling (MLM) has shown to be advantageous to pre-trained language models over the original single-token MLM, as entities/phrases and their dependencies…