Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text

Andrei Jarca, Florinel Alin Croitoru, Radu Tudor Ionescu

Masked language modeling has become a widely adopted unsupervised technique to pre-train large language models (LLMs). However, the process of selecting tokens for masking is rando…