109 citations · 164 across the 11 of their papers we have counts for
Showing 2020Show all
3 papers · 1 filter
cs.CL2020
Learning from Mistakes: Using Mis-predictions as Harm Alerts in Language Pre-Training
Chen Xing, Wenhao Liu, Caiming Xiong
Fitting complex patterns in the training data, such as reasoning and commonsense, is a key challenge for language pre-training. According to recent studies and our empirical observ…
cs.CL2020★ 4 cited
Taking Notes on the Fly Helps BERT Pre-training
Qiyu Wu, Chen Xing, Yatao Li +3
How to make unsupervised language pre-training more efficient and less resource-intensive is an important research direction in NLP. In this paper, we focus on improving the effici…
cs.LG2020★ 109 cited
On Layer Normalization in the Transformer Architecture
Ruibin Xiong, Yunchang Yang, Di He +7
The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is sh…