TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning

Yixuan Su, Fangyu Liu, Zaiqiao Meng +4

Masked language models (MLMs) such as BERT and RoBERTa have revolutionized the field of Natural Language Understanding in the past few years. However, existing pre-trained MLMs oft…