SAS: Self-Augmentation Strategy for Language Model Pre-training

Yifei Xu, Jingqiao Zhang, Ru He +4

The core of self-supervised learning for pre-training language models includes pre-training task design as well as appropriate data augmentation. Most data augmentations in languag…