Frustratingly Simple Pretraining Alternatives to Masked Language Modeling

Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina +1

Masked language modeling (MLM), a self-supervised pretraining objective, is widely used in natural language processing for learning text representations. MLM trains a model to pred…