9 citations · 14 across the 3 of their papers we have counts for
Showing cs.CLShow all
3 papers · 1 filter
cs.CL2020★ 9 cited
LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
Kaitao Song, Hao Sun, Xu Tan +4
While pre-training and fine-tuning, e.g., BERT~\citep{devlin2018bert}, GPT-2~\citep{radford2019language}, have achieved great success in language understanding and generation tasks…
cs.CL2020
MPNet: Masked and Permuted Pre-training for Language Understanding
Kaitao Song, Xu Tan, Tao Qin +2
BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models. Since BERT neglects dependency among predicted tokens, XLNet intr…
cs.CL2018
Hybrid Self-Attention Network for Machine Translation
Kaitao Song, Xu Tan, Furong Peng +1
The encoder-decoder is the typical framework for Neural Machine Translation (NMT), and different structures have been developed for improving the translation performance. Transform…