16 citations · 44 across the 7 of their papers we have counts for
10 papers · 1 filter
TranSFormer: Slow-Fast Transformer for Machine Translation
Bei Li, Yi Jing, Xu Tan +3
Learning multiscale Transformer models has been evidenced as a viable approach to augmenting machine translation systems. Prior research has primarily focused on treating subwords…
LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
Kaitao Song, Hao Sun, Xu Tan +4
While pre-training and fine-tuning, e.g., BERT~\citep{devlin2018bert}, GPT-2~\citep{radford2019language}, have achieved great success in language understanding and generation tasks…
MPNet: Masked and Permuted Pre-training for Language Understanding
Kaitao Song, Xu Tan, Tao Qin +2
BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models. Since BERT neglects dependency among predicted tokens, XLNet intr…
Representation Degeneration Problem in Training Natural Language Generation Models
Jun Gao, Di He, Xu Tan +3
We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the \emph{representation degeneration problem}. We obse…
Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
Hao Sun, Xu Tan, Jun-Wei Gan +4
Grapheme-to-phoneme (G2P) conversion is an important task in automatic speech recognition and text-to-speech systems. Recently, G2P conversion is viewed as a sequence to sequence t…
Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
Junliang Guo, Xu Tan, Di He +3
Non-autoregressive translation (NAT) models, which remove the dependence on previous target tokens from the inputs of the decoder, achieve significantly inference speedup but at th…