36 citations · 53 across the 12 of their papers we have counts for
Showing 2020Show all
2 papers · 1 filter
cs.CL2020★ 3 cited
Pre-Training a Language Model Without Human Language
Cheng-Han Chiang, Hung-yi Lee
In this paper, we study how the intrinsic nature of pre-training data contributes to the fine-tuned downstream performance. To this end, we pre-train different transformer-based ma…
cs.CL2020
Pretrained Language Model Embryology: The Birth of ALBERT
Cheng-Han Chiang, Sung-Feng Huang, Hung-yi Lee
While behaviors of pretrained language models (LMs) have been thoroughly examined, what happened during pretraining is rarely studied. We thus investigate the developmental process…