Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking

Mingyu Lee, Jun-Hyung Park, Junho Kim +2

Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations, but incurs substantial training costs. In this paper, we propose a nov…