1.2k citations · 1.7k across the 28 of their papers we have counts for
5 papers · 1 filter
Variance-reduced Language Pretraining via a Mask Proposal Network
Liang Chen
Self-supervised learning, a.k.a., pretraining, is important in natural language processing. Most of the pretraining methods first randomly mask some positions in a sentence and the…
Taking Notes on the Fly Helps BERT Pre-training
Qiyu Wu, Chen Xing, Yatao Li +3
How to make unsupervised language pre-training more efficient and less resource-intensive is an important research direction in NLP. In this paper, we focus on improving the effici…
MC-BERT: Efficient Language Pre-Training via a Meta Controller
Zhenhui Xu, Linyuan Gong, Guolin Ke +5
Pre-trained contextual representations (e.g., BERT) have become the foundation to achieve state-of-the-art results on many NLP tasks. However, large-scale pre-training is computati…
Rethinking Positional Encoding in Language Pre-training
Guolin Ke, Di He, Tie-Yan Liu
In this work, we investigate the positional encoding methods used in language pre-training (e.g., BERT) and identify several problems in the existing formulations. First, we show t…
Invertible Image Rescaling
Mingqing Xiao, Shuxin Zheng, Chang Liu +6
High-resolution digital images are usually downscaled to fit various display screens or save the cost of storage and bandwidth, meanwhile the post-upscaling is adpoted to recover t…