1.2k citations · 1.3k across the 11 of their papers we have counts for
6 papers · 1 filter
LazyFormer: Self Attention with Lazy Update
Chengxuan Ying, Guolin Ke, Di He +1
Improving the efficiency of Transformer-based language pre-training is an important task in NLP, especially for the self-attention module, which is computationally expensive. In th…
Revisiting Language Encoding in Learning Multilingual Representations
Shengjie Luo, Kaiyuan Gao, Shuxin Zheng +4
Transformer has demonstrated its great power to learn contextual word representations for multiple languages in a single model. To process multilingual sentences in the model, a le…
Variance-reduced Language Pretraining via a Mask Proposal Network
Liang Chen
Self-supervised learning, a.k.a., pretraining, is important in natural language processing. Most of the pretraining methods first randomly mask some positions in a sentence and the…
Taking Notes on the Fly Helps BERT Pre-training
Qiyu Wu, Chen Xing, Yatao Li +3
How to make unsupervised language pre-training more efficient and less resource-intensive is an important research direction in NLP. In this paper, we focus on improving the effici…
MC-BERT: Efficient Language Pre-Training via a Meta Controller
Zhenhui Xu, Linyuan Gong, Guolin Ke +5
Pre-trained contextual representations (e.g., BERT) have become the foundation to achieve state-of-the-art results on many NLP tasks. However, large-scale pre-training is computati…
Rethinking Positional Encoding in Language Pre-training
Guolin Ke, Di He, Tie-Yan Liu
In this work, we investigate the positional encoding methods used in language pre-training (e.g., BERT) and identify several problems in the existing formulations. First, we show t…