162 citations · 171 across the 5 of their papers we have counts for
4 papers · 1 filter
Masked Structural Growth for 2x Faster Language Model Pre-training
Yiqun Yao, Zheng Zhang, Jing Li +1
Accelerating large language model pre-training is a critical issue in present research. In this paper, we focus on speeding up pre-training by progressively growing from a small Tr…
Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation
Ziqi Wang, Yuexin Wu, Frederick Liu +5
Knowledge distillation is one of the primary methods of transferring knowledge from large to small models. However, it requires massive task-specific data, which may not be plausib…
EncT5: A Framework for Fine-tuning T5 as Non-autoregressive Models
Frederick Liu, Terry Huang, Shihang Lyu +3
Pre-trained encoder-decoder transformer architectures have become increasingly popular recently with the advent of T5 models. T5 has also become more favorable over other architect…
On the Transformer Growth for Progressive BERT Training
Xiaotao Gu, Liyuan Liu, Hongkun Yu +3
Due to the excessive cost of large-scale language model pre-training, considerable efforts have been made to train BERT progressively -- start from an inferior but low-cost model a…