Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model

Abstract

Masked Language Model (MLM) framework has been widely adopted for self-supervised language pre-training. In this paper, we argue that randomly sampled masks in MLM would lead to undesirably large gradient variance. Thus, we theoretically quantify the gradient variance via correlating the gradient covariance with the Hamming distance between two different masks (given a certain text sequence). To reduce the variance due to the sampling of masks, we propose a fully-explored masking strategy, where a text sequence is divided into a certain number of non-overlapping segments. Thereafter, the tokens within one segment are masked for training. We prove, from a theoretical perspective, that the gradients derived from this new masking schema have a smaller variance and can lead to more efficient self-supervised training. We conduct extensive experiments on both continual pre-training and general pre-training from scratch. Empirical results confirm that this new masking strategy can consistently outperform standard random masking. Detailed efficiency analysis and ablation studies further validate the advantages of our fully-explored masking strategy under the MLM framework.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• ERNIE: Enhanced Representation through Knowledge Integration
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• FinBERT: A Pretrained Language Model for Financial Communications
• FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
• Weighted Transformer Network for Machine Translation
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• Variance Reduction in SGD by Distributed Importance Sampling
• SciBERT: A Pretrained Language Model for Scientific Text
• A Proximal Stochastic Gradient Method with Progressive Variance Reduction
• Variance-reduced Language Pretraining via a Mask Proposal Network
• Train No Evil: Selective Masking for Task-Guided Pre-Training