PERT: Pre-training BERT with Permuted Language Model

Yiming Cui, Ziqing Yang, Ting Liu

Pre-trained Language Models (PLMs) have been widely used in various natural language processing (NLP) tasks, owing to their powerful text representations trained on large-scale cor…